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EXECUTIVE  SUMMARY 

Purpose.  To  support  the  Federal  Aviation  Administration’s 
(FAAs)  Air  Traffic  Organization  (ATO)  goal  to  reduce 
operational  errors,  we  were  asked  to  review  existing 
research  literature  on  operational  errors  (OEs)  so  that 
candidate  mitigation  strategies  could  be  identified  for 
immediate  deployment. 

Method.  This  report  covered  the  following  areas: 

•  A  brief  history  of  OE  investigation,  classification, 
and  reporting. 

•  Collection  of  154  OE-related  documents  covering 
a  45-year  span  from  1960-2005. 

•  Documentation,  review,  and  classification  of 222  OE 
prevention  and  reduction  initiatives  implemented 
from  1986  to  2005. 

•  Classification  of  all  materials  using  two  methods 

■  Type  of  study. 

■  HE  categorization  using  JANUS  taxonomy. 

•  Identification  of  knowledge  gaps  in  research  studies 
and  initiatives. 

•  Development  of  recommendations. 

Research  Results.  We  identified  several  findings  by 
analyzing  the  literature.  Some  consistent  research  find¬ 
ings  included: 

•  The  amount  of  traffic  measured  on  a  national  basis  is  the 
single  most  important  determinant  of  the  frequency  of 
OEs.  However,  while  the  amount  of  traffic  increased  at 
overall  in  the  NAS ,  the  amount  of  traffic  in  a  sector  when 
an  OE  occurs  has  generally  remained  unchanged. 

•  A  relatively  high  percentage  of  OEs  occurred  during 
the  first  20  minutes  on  position.  That  relationship  was 
consistent  across  options  and  during  much  of  the  day. 
While  this  result  was  often  linked  with  position  relief 
briefings,  only  a  small  percentage  of  OEs  were  attributed 
to  the  causal  factor  in  the  OE  reporting  form  related  to 
position  relief  briefings. 

•  Pilot/controller  miscommunications  were  historically 
identified  as  a  primary  causal  factor  associated  with  OEs 
and  hearback/readback  errors  were  studied  most  often. 
Although  analysis  of  recorded  communications  revealed 
that  few  hearback/readback  errors  resulted  in  an  OE, 
a  sizeable  proportion  of  OEs  were  attributed  to  hear- 
back/ readback  errors.  One  set  of  studies  found  a  strong 
relationship  between  the  complexity  of  the  controller’s 
transmission  and  the  probability  of  a  readback  error  in 
the  en  route  and  TRACON  environments. 


Less-frequent,  but  also  important  findings  include: 

•  One  study  found  a  statistically  significant  relation¬ 
ship  between  the  decline  in  supervisory  staffing  and 
the  overall  increase  in  OEs. 

•  Another  study  found  that  higher  percentages  of  OEs 
occurred  in  moderate  to  heavy  traffic  complexity 
conditions,  suggesting  that  the  interaction  of  the 
amount  of  traffic  with  traffic  complexity  may  affect 
OEs  more  than  the  amount  of  traffic  alone.  Other 
studies  also  identified  sector  complexity  as  a  factor 
contributing  to  OEs. 

•  Several  studies  have  found  that  controllers  with  five 
years  or  less  experience  (after  certification)  were  more 
susceptible  to  OEs  than  were  more  experienced 
controllers. 

•  Several  studies  related  OEs  to  problems  with  percep¬ 
tion  and  vigilance.  However,  a  lack  of  information 
about  these  factors  on  the  OE  reporting  form  pre¬ 
vented  an  in-depth  analysis  of  these  causal  factors. 

•  Although  memory  failures  were  associated  with  OEs 
in  all  ATC  environments,  much  of  the  memory 
literature  relevant  to  OEs  was  in  the  context  of 
runway  incursions.  Several  studies  found  memory 
failures  to  be  the  most  frequently  cited  causal  factor 
of  tower  OEs. 

Results  of  Initiatives.  Initiatives  related  to  organizational 
and  management  issues  primarily  dealt  with  develop¬ 
ment  and  review  of  national  and  local  QA  activities. 
Initiatives  described  concerns  about  resources  available 
to  supervisors  to  accomplish  their  jobs  and  recommended 
additional  supervisory  training.  Some  initiatives  focused 
on  mental  processes,  especially  those  efforts  addressing 
skills  training. 

Most  initiatives  introduced  by  the  air  traffic  organization 
that  related  to  contextual  conditions  involved  controller 
training,  teamwork,  and  communications.  Both  the  re¬ 
search  reports  and  OE  reduction  initiatives  emphasized  the 
same  six  contextual  conditions  (although  not  necessarily 
in  the  same  order):  a)  training  and  experience,  b)  team¬ 
work,  c)  pilot- ATC  communications,  d)  Human  Machine 
Interaction  (HMI)  and  equipment,  e)  airspace/surface, 
and  f)  traffic.  Although  the  specific  topics  addressed  by 
research  and  the  OE  reduction  interventions  appeared 
to  differ,  trend  comparisons  suggested  that  research  and 
operations  focused  on  the  same  general  areas. 
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We  also  identified  several  roadblocks  to  OE  reduction. 
These  are  briefly  noted  here  and  an  expanded  discussion 
is  presented  in  the  report. 

Initiatives:  The  legacy  Air  Traffic  Service  (ATS)  identi¬ 
fied  and  implemented  national,  regional,  and  facility 
level  OE  reduction  initiatives.  These  initiatives  ad¬ 
dressed  error  conditions  identified  through  analysis  of 
OE  reports.  While  the  reports  indicated  that  several 
of  these  initiatives  were  successful,  we  found  little 
documentation  to  support  those  claims.  Without  a 
method  to  track  the  effectiveness  ofinitiatives,  it  is  not 
possible  to  learn  from  past  successes  and  failures. 

Historical  OE  Database:  The  legacy  ATS  collected  an 
extensive  amount  of  information  about  OEs  during 
their  investigation  process  that  was  useful  for  guiding 
previous  OE  mitigation  strategies. 

•  Incomplete  data.  The  information  historically 
gathered  during  the  OE  investigation  process  did  not 
address  many  of  the  underlying  causal  dimensions 
typically  associated  with  human  error. 

•  N arrow  research  focus.  Despite  the  complexity  and 
dynamic  nature  of  a  controller’s  tasks,  most  research 
focused  on  only  one  or  two  factors.  For  example,  the 
conditions  of  the  complex  air  traffic  environment 
that  make  a  controller  more  vulnerable  to  making 
mistakes  versus  remaining  “error  free”  are  unclear. 
Furthermore,  objective  measures  of  controller  per¬ 
formance  besides  OEs  are  not  available. 

•  Similar  analyses.  Similar  analyses  have  been 
repeatedly  conducted  on  OE  data  that  resulted  in 
the  same  findings  as  previous  analyses  of  those  data. 
Analysts  must  be  encouraged  to  review  previous 
work  and  conduct  analyses  to  examine  how  multiple 
factors  interact  to  produce  OEs.  Cognitive  models 
of  controller  performance  should  be  used  to  guide 
these  analyses. 


Recommendations.  A  workgroup  consisting  of  members 
of  both  the  operational  and  research  communities  should 
be  convened  to  address  these  3  recommendations: 

1 .  Expand  OE  data  available  for  analysis 

a.  Improve  the  OE  data  collected  to  better  identify 
human  factors  associated  with  OEs. 

b.  Collect  baseline  data  about  normal  air  traffic 
operations  to  better  evaluate  conditions  sur¬ 
rounding  OEs. 

c.  Develop  objective  measures  of  controller  per¬ 
formance. 

2.  Improve  research  and  analyses 

a.  Account  for  the  interactive  nature  of  factors 
contributing  to  OEs. 

b.  Identify  high  impact  causal  factors  where  the 
payoff  for  mitigation  is  high. 

c.  Examine  OEs  relative  to  a  safety  culture. 

3.  Monitor  interventions 

a.  Develop  and  execute  a  method  for  tracking 
and  evaluating  OE  intervention  strategies. 

Conclusions.  Identification  of  OE  causal  factors  is  dif¬ 
ficult  because  they  occur  in  a  very  small  percentage  of 
ATC  operations.  This  review  has  shown  that,  historically, 
much  (sometimes  redundant)  research  was  conducted 
that  generated  little  new  information  about  why  OEs 
occurred.  Similarly,  many  initiatives  were  implemented, 
but  the  lack  of  a  systematic  follow-up  prevented  us  from 
learning  which  were  effective.  This  cycle  will  continue 
unless  steps  are  taken  to  1 )  obtain  relevant  data  to  conduct 
more  informative,  theory-based  analyses  and  2)  continu¬ 
ally  assess  the  effectiveness  of  OE  mitigation  strategies. 
The  ATO  must  monitor  research  efforts  and  operational 
initiatives  to  avoid  “reinventing  the  wheel”  (repeatedly 
conducting  the  same  analyses  and  initiatives)  and  “picking 
the  low  hanging  fruit”  (continuing  to  address  apparently 
easy  problems)  and,  thus,  wasting  increasingly  scarce 
resources.  The  ATO  must  develop  a  safety  culture  by 
obtainingbetter  data  about  the  circumstances  surrounding 
OEs  and  identifying  the  relative  importance  of  individual, 
supervisory,  and  organizational  contributions. 
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INTRODUCTION 

The  Federal  Aviation  Administration’s  (FAAs)  Air  Traffic 
Organization  (ATO)  has  a  five  year  performance  goal  of 
reducing  the  annual  number  of  serious  operational  errors 
(OEs)  from  a  total  of 637  in  2005  to  no  more  than  563  by 
the  end  of  2009,  equivalent  to  a  rate  of  3.18  per  million 
activities  (ATO,  2005).  An  OE  occurs  whenever  there  is 
a  violation  of  aircraft  separation  minima  that  is  the  result 
of  an  element  within  the  air  traffic  system  (e.g.,  a  facility 
procedure  or  an  air  traffic  control  specialist).  A  violation 
of  separation  minima  may  involve  a)  two  or  more  aircraft, 
b)  an  aircraft  and  terrain  or  obstacles,  or  c)  an  aircraft 
landing  or  departing  on  a  closed  runway  after  receiving 
air  traffic  authorization  to  do  so  (FAA,  2006R). 

The  ATO  seeks  to  implement  initiatives  to  reduce  OEs. 
To  best  develop  effective  OE  reduction  strategies,  it  is  im¬ 
portant  to  look  at  past  efforts  to  understand  and  mitigate 
OEs.  In  this  report,  we  seek  to  a)  review  and  classify  past 
research  reports  according  to  type  of  study  and  human 
factors  (HF)  categories,  b)  identify  HF  categories  not 
addressed  by  the  literature,  and  c)  review  previous  OE 
reduction  initiatives. 

We  first  describe  the  development  of  the  FAA  OE  inves¬ 
tigation  process  and  the  resulting  data  archive,  because 
the  data  collected  during  the  investigation  dictate  much 
of  what  is  known  about  OEs.  We  then  review  the  scien¬ 
tific  literature  regarding  OEs  and  describe  selected  OE 
reduction  initiatives.  Finally,  the  discussion  will  integrate 
information  from  the  review  of  the  scientific  literature 
and  the  OE  reduction  initiatives  to  identify  targets  of 
opportunity  for  further  research  and  mitigation  efforts. 

History  of  OE  Investigation, 

Classification,  and  Reporting 

This  section  describes  the  history  of  the  processes  used 
to  identify,  investigate,  classify,  and  report  OEs.  After 
describing  the  OE  investigation  and  classification  pro¬ 
cesses,  including  several  indices  used  to  rate  OE  severity, 
we  will  discuss  a  process  for  analyzing  the  causal  factors 
of  OEs,  called  JANUS,  which  will  be  used  to  classify  the 
OE  literature. 


OE  Identification 

For  a  number  of  years,  OEs  (then  called  system  errors) 
were  identified  through  pilot  reports,  supervisor  observa¬ 
tions,  reports  from  another  controller,  and  self-reports 
from  an  involved  controller.  This  led  to  considerable 
speculation  about  how  many  errors  actually  occurred 
and  how  severe  they  were. 

Following  the  initial  computerization  of  the  nation’s  en 
route  air  traffic  control  system,  the  FAA  embarked  on 
the  development  of  conflict  alert  software.  The  resulting 
conflict  alert  system  was  designed  to  notify  controllers  of 
an  eminent  loss  of  separation  between  aircraft.  On  Janu¬ 
ary  9,  1976,  the  conflict  alert  system  was  implemented 
at  the  20  en  route  air  traffic  control  centers  (ARTCCs) 
for  all  aircraft  flying  above  18,000  feet.  In  December 
1978,  the  system  was  modified  to  cover  all  aircraft  in 
en  route  airspace.  Meanwhile,  in  early  1978,  the  first 
Terminal  Radar  Approach  Control  (TRACON)  conflict 
alert  system  was  established  at  the  Houston  terminal.  By 
April  1978,  the  system  was  expanded  to  cover  62  facili¬ 
ties  (FAA,  1998b). 

During  the  mid  1980s,  the  Operational  Error  Detection 
Patch  (OEDP,  also  known  as  the  “snitch  patch”)  was 
introduced  in  en  route  facilities  to  alert  Area  Managers 
when  a  loss  of  standard  separation  occurred.  The  man¬ 
ager  then  determined  who  was  responsible  for  the  loss  of 
separation.  Thus,  at  en  route  facilities  today,  operational 
error  detection  and  reporting  is  automatic. 

In  TRACONs  and  towers,  error  detection  is  based  on 
either  self-reports  or  a  report  from  another  controller, 
supervisor,  or  pilot.  The  Department  of  Transportation 
Office  of  the  Inspector  General  (OIG)  issued  a  report 
(DOT,  2003)  expressing  concern  over  the  potential 
for  underreporting  of  errors  at  facilities  where  OEs  are 
self-reported.  ATO  shares  this  concern  and  is  working 
toward  an  automated  system  for  the  TRACONs  that 
will  help  detect  OEs.  However,  for  now,  the  variability 
in  OE  reporting  (and  presumed  underreporting  in  the 
terminal  environments)  needs  to  be  taken  into  account 
when  interpreting  available  OE  data. 
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OE  Investigation  Process 

An  approach  for  documenting  and  classifying  air  traffic 
control  errors  was  developed  in  1965  and  described  in 
FAA  Order  8020.  The  Air  Traffic  Service  (ATS)  System 
Error  Reporting  Program  established  a  formal  process 
to  investigate  and  classify  what  were  then  called  system 
errors.  As  O’Connor  &  Pearson  (1965)  indicated,  a  re¬ 
porting  program  for  system  errors  should  recognize  the 
dynamic  and  complex  nature  of  the  air  traffic  system.  It 
should  take  into  account  issues  associated  with  equipment, 
personnel,  procedures,  and  aspects  of  decision-making 
that  influence  performance.  Since  that  time,  periodic 
attempts  have  been  made  to  update  the  OE  reporting 
form  to  ensure  that  sufficient  information  is  gathered 
to  assess  system  safety  and  identify  error-prone  events, 
conditions,  and  procedures  that  could  be  modified  to 
mitigate  OEs. 

Currently,  when  an  OE  occurs,  a  two-stage  investigatory 
process  is  initiated  consisting  of  a  preliminary  and  final 
investigation.  As  specified  in  FAA  O rder  7210. 56  (2002), 
an  employee  who  is  aware  that  an  OE  may  have  occurred 
must  immediately  report  the  occurrence  to  any  available 
supervisor  or  Controller-in-Charge  (CIC).  After  the  OE 
has  been  reported,  the  area  supervisor  will  conduct  a 
preliminary  fact-finding  exercise  to  determine  the  valid¬ 
ity  of  the  suspected  OE.  If  validated,  the  area  supervisor 
follows  the  procedures  listed  in  FAA  Order  7210.56  and 
records  the  information  on  the  preliminary  investigation 
report  (Form  7210-2),  which  includes  a  limited  checklist 
of  OE  causal  factors.  After  the  preliminary  investigation 
report  is  submitted,  air  traffic  management  (ATM)  at  the 
facility  determined  to  be  responsible  for  the  OE  desig¬ 
nates  an  Investigator-In-Charge  (IIC)  to  conduct  a  more 
thorough  analysis  of  the  factors  that  contributed  to  the 
OE.  The  results  of  the  IIC’s  investigation  are  detailed  in 
a  final  investigation  report,  which  includes  a  checklist  of 
OE  causal  factors  comprising  data  posting,  radar  display, 
aircraft  observation  (at  towers),  communication,  coordi¬ 
nation,  and  position  relief  briefings.  In  addition  to  the 
information  provided  by  the  IIC,  the  manager  may  also 
offer  comments  about  what  may  have  led  to  the  error. 

An  OE  maybe  associated  with  more  than  one  causal  factor. 
Based  on  data  from  the  final  investigation  report,  an  OE 
Severity  Index  (SI)  score  (described  below)  is  calculated 
by  the  ATO-Safety  Office  of  Investigations  using  a  stan¬ 
dardized  methodology  and  included  in  the  final  report. 
OEs  in  categories  A  (high  severity)  and  B  (high  moderate 
severity)  are  subjected  to  greater  organizational  scrutiny 
than  those  in  categories  C  (low-moderate  severity)  and 
D  (low  severity). 


OE  Classification 

Simpson  (1982)  was  one  of  the  first  individuals  to  attempt 
to  classify  the  risk  associated  with  OEs.  He  defined  three 
collision  risk  zones  based  on  the  amount  of  vertical  and 
horizontal  separation  between  involved  aircraft.  A  “near 
miss”  was  considered  to  occur  if  two  aircraft  passed  within 
500  feet  in  both  directions.  The  near  miss  had  subzones 
of  “critical  near  miss”  (within  100  feet)  and  “potential 
near  miss”  (between  100  and  500  feet).  From  the  edge 
of  the  near  miss  zone  to  1,000  feet  vertical  and  3,000 
feet  horizontal  separation  was  designated  as  the  “near 
approach  zone.”  The  last  zone,  called  the  “separation 
infringement”  zone,  extended  to  the  limits  of  vertical  and 
horizontal  separation  that  apply  to  the  airspace. 

As  part  of  a  subsequent  OE  investigation  report  (FAA, 
1985),  the  Air  Traffic  Quality  Assurance  Staff  introduced 
a  system  that  classified  errors  as  major,  moderate,  or  mi¬ 
nor,  based  on  the  amount  of  separation  loss.  Points  were 
assigned  for  the  degree  of  loss  of  horizontal  (10  points) 
and  vertical  separation  (10  points).  A  separate  rating  was 
provided  for  nonradar/Oceanic  positions.  A  major  error 
(20  points)  involved  less  than  Vi  mile  horizontal  separation 
and  less  than  500  feet  vertical  separation.  Moderate  er¬ 
rors  involved  several  different  combinations  of  horizontal 
and  vertical  separation  loss  (14-19  points).  An  example 
would  be  2  to  less  than  2  Vi  miles  of  horizontal  separation 
(5  points)  and  500  feet  to  less  than  600  feet  of  vertical 
separation  (9  points),  for  traffic  under  flight  level  (FL) 
290.  Minor  errors  were  assigned  between  1-13  points. 
The  total  points  used  to  classify  severity  for  operations 
requiring  less  than  3-mile  separation  (i.e.,  in  TRACON 
airspace)  were  slightly  different.  This  process  was  only 
used  until  November  of  1988. 

In  their  initial  report,  the  FAA  Office  of  Aviation  Safety 
(ASF)  described  a  measure  of  potential  risk  using  prox¬ 
imity  (also  called  Root  Mean  Square,  RMS,  distance; 
Rodgers,  Mogford,  &  Mogford,  1998).  The  proximity 
metric  was  calculated  by  computing  the  square  root  of 
the  sum  of  the  squares  of  the  minimum  horizontal  and 
vertical  separations  between  the  aircraft  involved.  This 
is  equivalent  to  the  Euclidean  distance  between  the  two 
involved  aircraft.  Analysis  of  this  measure  suggested  that 
the  vast  maj  ority  of  OEs  occurred  at  5,000  feet  of  proxim¬ 
ity  or  greater.  This  was  true  for  ARTCCs,  TRACONs, 
and  control  towers  (FAA,  1988). 

OE  Severity  Index 

In  200 1 ,  an  assessment  of  the  safety  risk  associated  with 
OEs  was  added  to  the  OE  investigation  process.  The  100- 
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point  SI  is  based  on  four  categories:  Category  A  -  high 
severity,  Category  B  -  high  moderate  severity,  Category 
C  -  low  moderate  severity,  and  Category  D  -  low  severity. 
Classification  of  severity  is  based  on  the  following  fac¬ 
tors:  vertical  and  horizontal  separation  distances,  relative 
flight  paths,  cumulative  closure  rates,  and  the  level  of 
ATC  control.  Because  most  of  these  factors  are  based  on 
operationally  relevant  criteria  associated  with  providing 
ATC  services,  the  IIC  is  typically  able  to  calculate  the 
severity  of  ATC  separation  losses  during  the  OE  investi¬ 
gation.  However,  personnel  assigned  to  the  ATO  Safety 
Office  at  FAA  headquarters  are  responsible  for  making 
a  final  decision  regarding  the  severity  of  each  OE.  As 
noted  earlier,  the  ATO  uses  Category  A  and  B  events  as 
organizational  performance  metrics. 

RISC  Model 

The  Runway  Incursion  Severity  Categorization  (RISC) 
model  is  an  automated  approach  to  categorizing  the 
severity  of  the  outcome  of  runway  incursions  (RIs) 
whether  they  are  classified  as  OEs,  pilot  deviations,  or 
vehicle/pedestrian  deviations. The  RISCmodel  (Cardosi, 
Hannon,  Sheridan,  &  Davis,  2005)  reflects  the  likelihood 
that  an  RI  could  have  resulted  in  a  collision.  The  paths 
of  the  aircraft  or  vehicles  involved  in  an  RI  are  reviewed, 
and  the  closest  horizontal  and  vertical  proximity  is  de¬ 
termined.  The  model  also  takes  into  account  avoidance 
maneuvers,  visibility  at  the  time  of  the  RI,  the  types  of 
errors  made  by  the  pilot  and  controller,  and  the  time 
available  for  the  pilot  to  respond.  Cardosi  pointed  out 
that  the  RISC  model  should  not  be  considered  as  a  rating 
of  “the  adequacy  of  the  ATC  services  or  the  severity  of 
the  operational  (OE)  that  resulted  in  the  incursion  (pg. 
2).”  The  levels  of  risk  of  collision  range  from  Category 
D,  which  have  little  or  no  chance  of  collision  but  met 
the  definition  of  a  runway  incursion,  to  Category  A,  in 
which  a  collision  is  narrowly  avoided.  Events  are  assigned 
to  these  categories  based  on  five  operational  definitions: 
a)  available  reaction  time,  b)  the  need  for  evasive  or  cor¬ 
rection  action,  c)  environmental  conditions,  d)  aircraft 
or  vehicle  speed,  and  e)  proximity. 

The  goal  of  the  automated  rating  model  was  to  analyze 
recorded  events  and  identify  OE  severity  in  a  way  that 
resembled  the  expertise  of  a  group  of  Subject  Matter  Ex¬ 
perts  (SMEs).  Cardosi  et  al.  (2005)  conducted  an  initial 
validation  study  that  compared  the  ratings  generated 
by  the  model  with  ratings  provided  by  SMEs  from  the 
Office  of  Runway  Safety  and  Operational  Services.  The 
ratings  of  the  model  matched  the  group’s  ratings  in  67% 
of  the  cases.  However,  when  the  C  &  D  categories  were 
combined,  there  was  only  a  5%  disagreement  between  the 


model  and  the  SMEs.  The  model  has  since  been  refined 
and  FAA  continues  the  validation  process. 

OE  Reporting 

Recording  of  OE  investigational  data  began  when  the 
investigation  process  was  initiated  in  1965.  In  1984,  the 
Office  of  Aviation  Safety  (ASF)  designed  the  National 
Airspace  Information  Monitoring  System  (NAIMS)  and 
implemented  it  in  1985.  One  element  of  the  larger  auto¬ 
mated  NAIMS  database  is  the  Operational  Error  System 
(OES),  which  includes  information  derived  from  the  OE 
reports,  including  possible  causal  factors.  Since  1985,  as 
part  of  their  safety  oversight,  ASF  has  prepared  annual 
reports  that  analyze  the  OES.  The  reports  describe  many 
variables  of  interest  with  regard  to  OEs:  overall  error  rate, 
error  rates  by  facilities,  causal  factors,  and  characteristics 
of  errors/error  rates  by  facility  (FAA,  1988). 

The  form  used  to  obtain  information  about  the  back¬ 
ground  and  causes  of  OEs  has  changed  over  the  years, 
and  the  database  containing  that  information  has  been 
maintained  by  different  organizations.  Changes  in  the 
form  necessarily  result  in  changes  to  variables  included 
in  the  OES  database. 

OE  Research  Considerations 

An  issue  that  must  be  considered  when  conducting  re¬ 
search  on  OES  data  is  that  some  data  fields  recorded  on 
the  forms  have  changed  over  the  years  as  AT  investiga¬ 
tors,  analysts,  and  managers  changed  their  information 
requirements.  For  this  reason,  any  analyses  of  the  database 
should  include  checks  to  ensure  that  findings  are  based 
on  characteristics  of  the  underlying  data  rather  than 
reflecting  changes  made  to  the  form.  Reports  resulting 
from  these  analyses  should  indicate  the  authors’  actions 
to  minimize  errors  resulting  from  changes  to  informa¬ 
tion  in  data  fields.  An  assessment  is  currently  underway 
to  document  when  changes  to  the  form  (and  thus  to  the 
database)  occurred.  Similarly,  checks  should  be  imple¬ 
mented  to  identify  data  entry  errors. 

Another  issue  associated  with  research  in  this  area  is  that, 
although  several  previous  attempts  have  been  made  to 
quantify  the  level  of  risk  associated  with  an  OE,  most  of 
the  reports  included  in  this  review  (especially  studies  that 
examined  the  OES  database)  involved  OEs  that  were  stud¬ 
ied  without  regard  to  the  associated  level  of  severity.  Thus, 
the  nature  of  the  OE  review  and  classification  process  limits 
data  analysis  and  constrains  interpretation  of  the  results. 
Certain  questions  about  HF  issues  cannot  be  addressed 
within  the  limitations  of  the  existing  database. 
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METHOD 

This  section  describes  our  classification  of  the  OE  reports 
reviewed  for  this  project.  First,  the  documents  we  acquired 
are  described,  followed  by  a  discussion  of  two  dimensions 
used  to  classify  the  documents:  a)  type  of  study  conducted 
andb)  an  HE  OE  classification  method.  Categories  within 
each  dimension  are  described  below. 

Source  Documents 

One  hundred  fifty-nine  OE-related  documents  were 
collected,  covering  a  45-year  span  from  1960-2005.  We 
searched  six  aviation  technical  report  databases  to  identify 
the  relevant  literature,  using  key  words  included  in  the 
phrase  “air  traffic  control  operational  errors.”  The  six 
databases  included:  a)  the  European  Organization  for 
the  Safety  of  Air  Navigation  Report  Database,  b)  the 
University  of  Illinois  Institute  of  Aviations  Aviation  Re¬ 
search  Laboratory  Report  Database,  c)  the  WJ.  Hughes 
Technical  Center  Technical  Reports  Database,  d)  the 
Civil  Aerospace  Medical  Institute  (CAMI)  Technical 
Report  Database,  e)  the  National  Aeronautics  and  Space 
Administration  (NASA)  Technical  Report  Database,  and 
f)  the  National  Transportation  Safety  Board  (NTSB)  Re¬ 
port  Database.  Ovid  software  was  also  used  to  search  the 
scientific  literature  contained  in  the  American  Psychologi¬ 
cal  Association’s  PsychINFO®  database.  PsychINFO® 
is  a  comprehensive  international  bibliographic  database 
of  psychology  that  contains  citations  and  summaries  of 
peer-reviewed  journal  articles,  book  chapters,  books. 


dissertations,  and  technical  reports  written  in  the  field  of 
psychology  and  other  reports  addressing  the  psychological 
aspects  of  related  disciplines.  Proceedings  from  scientific 
sessions  were  also  examined,  as  were  unpublished  internal 
FAA  documents  acquired  through  a  network  of  profes¬ 
sional  contacts  within  the  ATO.  (To  date,  there  is  no 
centralized  repository  of  internal  FAA  reports  describing 
the  implementation  or  evaluation  of  OE  interventions.) 
We  included  internal  FAA  documents  in  this  report  if 
they  addressed  causal  factors  of  OEs  and  not  those  solely 
focused  on  ATC  or  aviation  safety. 

Figure  1  shows  the  distribution  over  time  of  documents 
included  in  the  literature  review.  The  increase  in  the 
number  of  research  reports  published  after  1985  may  be 
due,  in  part,  to  an  interest  in  assessing  the  recovery  and 
safety  of  the  NAS  following  the  ATC  strike  and  resulting 
loss  of  11,500  controllers  in  August  1981. 

Although  the  graph  suggests  that  most  studies  were 
conducted  since  1980,  it  should  also  be  noted  that  it  was 
difficult  to  identify  and  obtain  information  about  studies 
conducted  before  that  time.  The  difficulty  stemmed  from 
the  lack  of  a  central  repository  for  OE  research  reports.  It 
is  possible  that  early  studies  were  conducted  but  were  not 
archived  for  future  reference.  Thus,  we  may  find  ourselves 
initiating  OE  studies  to  rediscover  what  once  was  known 
but  has  been  lost  from  institutional  knowledge.  Without 
an  ongoing  systematic  means  for  archiving  OE  research 
reports,  as  they  are  made  available,  the  institutional 
memory  of  past  work  is  lost. 


Years 

Figure  1 .  Distribution  of  OE-related  documents  by  year  of  publication. 
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Classification  Procedures 

Each  document  was  reviewed  and  classified  along  two 
dimensions.  The  first  was  the  type  of  study  conducted. 
The  second  involved  a  comparison  of  the  content  with 
a  set  of  human  factors-related  categories  included  in  a 
recently  developed  OE  analysis  method  called  JANUS. 
Definitions  of  all  classification  categories  are  provided 
below.  The  results  of  the  classification  processes  appear 
in  Appendices  A  and  B. 

Type  of  study 

Studies  were  classified  into  one  of  four  types:  Descriptive 
studies  (database  analysis),  theory-based  studies,  field 
studies,  and  experiments.  This  classification  system  was 
based  on  the  way  scientific  inquiries  are  conducted — de¬ 
scribe  the  phenomenon  of  interest,  then  develop  a  theory 
to  understand  what  has  been  observed,  then  conduct 
field  studies  and  experiments  to  better  understand  the 
mechanisms  that  affect  the  phenomenon.  Each  approach 
has  advantages  and  disadvantages,  depending  on  the  goal 
of  the  study.  Each  report  was  classified  into  only  one 
category  based  on  the  major  thrust  of  the  article.  These 
were  defined  as  follows: 

Descriptive  study  (database  analysis) 

A  report  was  classified  in  this  group  if  it  described  informa¬ 
tion  contained  in  the  OE  database  and  did  not  necessarily 
test  hypotheses.  This  category  also  includes  exploratory 
studies  (i.e.,  database  analysis)  performed  on  OE  data  to 
try  to  discover  underlying  trends  and  associations  between 
variables  of  interest  (e.g.,  association  between  the  amount 
of  traffic  activity  and  the  number  of  OEs). 

Theory-based  study 

A  report  was  classified  in  this  group  if  it  presented  and 
interpreted  information  about  OEs  in  the  context  of  a 
particular  way  of  thinking  (e.g.,  interpreted  OE  causal 
factors  from  an  information  processing  perspective, 
from  a  perspective  based  on  previous  experience,  or  ac¬ 
cording  to  a  predetermined  organizing  rationale  [i.e.,  a 
taxonomy]).  Using  this  definition,  a  theory-based  report 
can  range  from  a  “thought  piece”  in  which  the  author(s) 
speculates  about  cause  and  effect  mechanisms  based  on 
his/her  knowledge  of  the  issue  to  a  more  formal  model 
based  on  the  results  of  empirical  research.  Examples  of 
the  former  would  be  a  “lessons  learned”  report  written 
by  a  former  certified  professional  controller  (CPC)  or  the 
reflections  of  an  HE  researcher  who  has  spent  a  career 
studying  OEs.  An  example  of  the  latter  would  be  using  a 


human  error  taxonomy  (e.g.,  JANUS)  to  organize  data  so 
that  additional  insight  can  be  gained  about  the  OE. 

Field  Study 

A  report  was  classified  in  this  group  if  it  described  a  study 
conducted  under  actual  job  conditions,  although  the 
nature  of  the  tasks  being  observed  and  data  being  col¬ 
lected  may  vary.  An  example  would  be  a  time  and  motion 
study  of  actual  ATC  equipment  usage  conducted  while 
controllers  are  actively  controlling  traffic,  rather  than 
participating  in  AT C  simulations.  Another  example  would 
be  analyzing  routinely  recorded  ATC  data  to  establish  a 
baseline  of  operations  before  a  new  piece  of  equipment 
is  introduced.  Because  the  data  collections  are  conducted 
in  actual  working  conditions,  it  is  often  difficult  to  ma¬ 
nipulate  the  variables  of  interest  to  make  cause  and  effect 
determinations.  Thus,  most  ATC  field  studies  fall  under 
the  category  of  observational  research  or  baseline/  trend 
analyses  and  are  often  used  to  assess  the  acceptability  of 
changes  in  equipment  and/or  procedures. 

Experiment 

A  report  was  classified  in  this  group  if  it  presented  a  study 
conducted  in  an  experimental  setting,  such  as  an  ATC 
simulation,  and  used  the  scientific  method  to  investi¬ 
gate  the  effects  of  independent  variables  (i.e.  predictor 
variables  such  as  situation  awareness,  often  called  SA;  or 
working  memory)  on  OE  occurrence  (the  criterion  of 
interest).  When  the  scientific  method  is  strictly  applied, 
experiments  enable  researchers  to  control  the  experimen¬ 
tal  setting,  allowing  measurement  of  the  direct  effects 
of  predictor  variables  on  the  criteria.  In  other  words, 
experiments  enable  researchers  to  test  for  cause-and-ef- 
fect  relationships. 

JANUS  as  an  Organizing  Framework 

The  current  OE  reporting  process  produces  enormous 
amounts  of  data  about  each  OE.  The  resulting  OE  reports 
are  largely  descriptive  of  the  circumstances  involved  in 
the  events  and  do  not  provide  a  systematic  approach  to 
understanding  the  underlying  causal  human  factors.  When 
we  began  our  literature  review,  we  needed  a  method  to 
organize  the  large  number  of  OE  reports  identified  in 
the  literature  search.  Many  of  these  reports  are  based  on 
analyses  of  the  OES  database. 

To  organize  the  OE  reports,  we  borrowed  the  taxonomy 
from  JANUS  (Pounds  &  Isaac,  2002, 2003),  a  technique 
that  integrated  two  aviation  error  classification  systems: 
a)  the  Human  Factor  Analysis  and  Classification  System, 
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JANUS  -  A  Joint  A  nalysiis  System  for  ATC 
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Figure  2.  UANUS,  a  comprehensive  HF  framework  for  analyzing  OEs. 


(HFACS;Shappell&Wiegmann,  2000b),  and  b)  Human 
Error  Reduction  in  Air  Traffic  Management,  (HERA; 
EATMP,  2000)  to  enhance  identification  of  OE  causal 
factors.  JANUS  provides  an  inclusive  HF  framework 
that  captures  an  extensive  array  of  potential  OE  causal 
factors  (see  Figure  2).  Using  the  JANUS  taxonomy,  the 
human  factors  associated  with  an  OE  can  be  classified 
in  terms  of  the  person  performing  a  specific  task  with  a 
particular  piece  of  equipment  in  a  specific  environment, 
extending  to  supervisory  and  organizational  influences 
that  may  affect  the  person’s  performance.  Using  the  cat¬ 
egories  and  factors  from  this  technique  provided  us  with 
a  comprehensive  framework  for  classifying  the  reports 
included  in  the  literature  review. 

Although  originally  developed  to  analyze  ATC  OEs,  the 
JANUS  framework  has  two  useful  properties  that  we 
exploited  to  organize  the  OE  literature.  First,  the  JANUS 
taxonomy  enabled  the  literature  to  be  classified  into 
several  HF-related  areas.  Thus,  by  simply  classifying  the 
literature  according  to  the  JANUS  categories,  one  can  see 
which  human  factors-related  areas  (e.g.,  mental  process¬ 
ing,  contextual  conditions,  supervision,  or  organizational 
influences)  have  received  the  most  attention  and  which 
have  been  neglected.  Second,  when  the  type  of  study 
conducted  (i.e.,  experiment,  field  study,  descriptive,  or 
theoretical)  is  used  to  classify  the  literature,  it  is  possible 
to  identify  the  amount  of  scientific  rigor  that  has  been 
dedicated  to  a  given  human  factors-related  area. 


When  classifying  the  literature  using  the  JANUS  tax¬ 
onomy  (see  Appendix  B),  topical  categories  (defined  by 
factors)  were  used  instead  of  formal  definitions.  When 
a  report  addressed  several  important  research  topics, 
multiple  JANUS  categories  were  used  to  classify  it.  The 
JANUS  categories  were  defined  as  follows: 

Organizational  Influences 

A  report  was  classified  in  this  group  if  it  described  organi¬ 
zational  factors  contributing  to  OEs  (e.g.,  organizational 
climate,  structure,  national  or  regional  policies  and  pro¬ 
cedures,  and/or  resource  management).  Organizational 
issues  involve  FAA  Headquarters,  Service  Areas  (formerly 
called  FAA  Regions),  etc. 

Management 

A  report  was  classified  in  this  group  if  it  described 
management  factors  contributing  to  OEs  (e.g.,  facility 
policies  and  procedures,  general  management  planning, 
correction  of  known  problems).  Management  issues 
involve  facility  managers,  facility  Operations  Managers 
(OMs),  and  staff 

Supervision 

A  report  was  classified  in  this  group  if  it  described  fac¬ 
tors  related  to  controllers’  operational  supervisors  and 
discussed  how  they  might  contribute  to  OEs  (e.g. ,  general 
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supervision,  planning,  correcting  known  problems,  team¬ 
work,  and  supervisory  non-compliance). 

Contextual  Conditions 

A  report  was  classified  in  this  group  if  it  described  the 
context  in  which  OEs  occurred.  The  12  categories  of 
contextual  conditions  used  were  traffic/ airspace,  weather, 
pilot  actions,  pilot/ATC  communications,  procedures, 
documents/materials,  ambient  environment,  teamwork, 
individual/personal  conditions,  interpersonal  and  social 
conditions,  training  and  experience,  and  Human-Machine 
Interaction  (HMI)/Equipment. 

Non-compliance^  Issues 

A  report  was  classified  in  this  group  if  it  addressed  pur¬ 
poseful  disregard  of  rules  or  procedures. 

Task  Descriptions 

A  report  was  classified  in  this  group  if  it  described  ac¬ 
tivities  in  which  the  controller  was  primarily  engaged  at 
the  time  of  an  OE,  such  as  communicating  with  a  pilot, 
control  room  communication,  radar  monitoring,  tower 
observation,  giving  a  position  relief  briefing. 

Response  Execution 

Response  execution  refers  to  the  outputs  or  actions  based 
on  the  earlier  processes,  that  is,  how  choices  and  plans 
were  enacted  as  actions  (either  verbally  or  manually). 

Mental  Processes 

A  report  was  classified  in  this  group  if  it  described 
controllers’  thought  processes  associated  with  OEs. 
These  processes  included:  a)  perception  and  vigilance, 
b)  memory,  c)  planning  and  decision-making,  and  d) 
response  execution. 

RESULTS 

The  results  are  organized  into  two  sections.  The  first 
describes  the  classification  of  OE  reports  according  to 
type  of  study.  We  follow  with  the  coding  of  the  literature 
according  to  the  HE  categories  contained  in  JANUS.  A 
description  is  provided  for  some  of  the  studies  contained 


’  In  the  initial  work  with  EUROCONTROL,  the  term  “contravention” 
was  used  instead  of  “non-compliance.  ”  When  FAA  air  traffic  controllers 
reviewed  the  JANUS  materials,  they  preferred  to  use  the  term  “non- 
compliance”  for  this  concept. 


in  the  JANUS  categories  that  were  most  frequently 
addressed  by  research. 

The  second  section  describes  available  information  about 
AT  initiatives  that  have  been  developed  nationally,  region¬ 
ally,  and  at  individual  facilities  to  reduce  operational  errors. 
Once  again,  we  review  these  interventions  as  they  relate 
to  the  various  JANUS  categories. 

Literature  Review 

Classifying  the  Literature  by  Type  of  Study 

Appendix  A  lists  the  research  documents  classified  ac¬ 
cording  to  the  type  of  study  conducted.  The  distribution 
showing  the  number  of  each  type  of  study  is  shown  in 
Figure  3.  The  majority  of  documents  collected  were 
classified  as  descriptive  studies  (database  analysis;  44%), 
followed  by  theoretical  studies  (36%).  Looking  at  the 
distribution  in  Figure  3,  it  is  apparent  that  the  more 
scientifically  rigorous  HE  methods  (experiments,  9%, 
and  field  studies,  11%)  have  seldom  been  employed  to 
study  OEs  and  their  causes. 

Descriptive  Studies  (Database  Analysis) 

Given  that  most  of  the  empirical  information  available 
about  OEs  and  their  causes  has  come  from  descriptive 
analyses,  it  is  imperative  to  examine  that  literature  with 
an  eye  toward  the  quality  of  the  information  contained 
within  the  respective  OE  databases. 

Eight  databases  that  contain  OE-related  information  were 
referenced  in  the  descriptive  literature.  These  were  the: 
a)  National  Airspace  Information  Monitoring  System 
(NAIMS),  b)  NASA’s  Aviation  Safety  Reporting  System 
(ASRS),  c)  NTSB’s  Aviation  Accident  and  Incident  Data 
System,  d)  AT  Quality  Assurance  Analysis  and  Reporting 
(QUASAR)  database,  e)  the  FAAs  System  Effectiveness 
Information  System  (SEIS)  database,  f)  FAAs  Operational 
Errors  System  (OES)  database  developed  from  prelimi¬ 
nary  and  final  OE  reports,  g)  OE  databases  at  individual 
FAA  facilities,  and  h)  the  non-specific  archival  FAA  OE 
database.  Several  of  these  databases  are  based  on  earlier 
versions  of  the  current  OES  database.  Although  it  is 
beyond  the  scope  of  this  report  to  describe  each  of  these 
databases  in  detail,  it  is  important  to  note  that  there 
is  no  centralized  quality  assurance  (QA)  process  that 
governs  the  quality  of  the  data  in  each  of  the  respective 
databases.  This  becomes  especially  problematic  when 
trying  to  compare  the  results  of  OE  studies  that  extracted 
data  from  different  databases.  For  example,  how  might 
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Figure  3.  Distribution  of  OE-related  documents  by  type  of  study. 


one  equate  studies  that  used  information  from  the  ASRS 
(which  includes  only  self-reports  of  incidents)  or  from 
a  generic  FAA  OE  database  at  a  field  facility  (which 
contains  only  data  specific  to  that  facility)  with  studies 
that  used  information  from  the  NAIMS  (which  contains 
information  about  OEs  from  all  FAA  facilities)? 

Theory-based  Reports 

The  authors  of  theory-based  reports  apply  human  factor 
principles  and  models  of  human  error  to  speculate  about 
the  underlying  causes  of  OEs.  For  example,  Reason’s 
(1990)  model  and  Rasmussen’s  (1986)  information 
processing  model  both  propose  generic  models  of  hu¬ 
man  error.  The  merit  of  using  generic  models  of  human 
error  to  understand  the  underlying  HF  causes  of  OEs 
was  discussed  as  early  as  1977.  Kinney  (1977c)  noted 
that  to  some  degree,  all  OEs  involve  the  basic  human 
failings  of  memory,  attention,  and  judgment,  regardless 
of  the  situation.  Thus,  by  developing  interventions  to 
address  common  human  error  vulnerabilities  in  general, 
the  ATC  system  should  experience  an  overall  decline  in 
the  number  of  OEs.  However,  one  of  the  shortcomings 
of  this  line  of  thinking  is  that  generic  models  assume 
that  humans  are  interchangeable.  The  problem  comes 
when  attempts  are  made  to  apply  generic  models  to  ad¬ 
dress  specific  events  surrounding  an  OE.  Generic  models 
produce  generic  solutions  and  must  be  customized  to 
the  individual  controllers/ facilities/sectors  to  solve  a 
particular  problem. 


Most  of  the  theory-based  reports  we  reviewed  served  as 
heuristics  for  designing  and  conducting  HF  experiments 
and/or  field  studies,  rather  than  addressing  a  particular 
operational  problem  or  recommending  strategies  for  er¬ 
ror  mitigation.  However,  the  recommended  experiments 
and/or  field  studies  were  seldom  completed. 

Field  Studies 

In  the  field,  it  is  neither  possible  nor  desirable  to  manipu¬ 
late  working  conditions  to  assess  their  influence  on  the 
likelihood  of  committing  an  OE.  For  that  reason,  none  of 
the  field  studies  identified  for  this  literature  review  directly 
examined  the  causes  of  OEs  but,  instead,  addressed  OE 
precursors  such  as  traffic  volume  (Hurst&Rose,  1978a), 
sector  complexity  (Hurst  and  Rose,  1978b;  Grossberg, 
1989),  problematic  communications  (Burki-Gohen, 
1995;  Gardosi,  1993;  Gardosi,  1994;  Gardosi,  Brett, 
&  Han,  1996;  Prinzo,  1996),  SA  (Endsley  &  Rodgers, 
1994),  and  the  ATG  culture  (Jones,  1997).  There  is  a 
large  body  of  literature  that  addresses  many  of  these  and 
other  HF  issues  where  the  emphasis  is  more  about  the 
effects  of  these  factors  on  performance  rather  than  on  the 
occurrence  of  OEs.  While  these  investigations  provided 
us  with  greater  insight  into  some  of  the  complexities 
associated  with  the  operational  ATG  environment,  they 
provided  little,  if  any,  information  about  the  linkage  of 
these  factors  to  OEs. 


An  exception  to  the  typical  limitations  associated  with 
field  studies  was  the  joint  FAA-EUROCONTOL  project 
conducted  by  Pounds  and  Isaac  (2003)  to  test  the  JANUS 
technique.  The  FAA’s  test  of  the  technique  encompassed 
twenty-nine  ATC  facilities  which  volunteered  to  collect 
JANUS  data  in  parallel  with  the  FAA  OE  investigation 
process.  This  represented  a  concurrent  test  of  the  JANUS 
OE  analysis  process  because  researchers  interviewed  ATC 
personnel  to  collect  JANUS  data  while  the  facility’s  staff 
conducted  their  investigation  per  FAA  Order  7210.56. 
The  technique  was  evaluated  first  based  on  the  feasibility 
of  the  process  and  then  for  the  information  value  of  the 
data  collected. 

When  asked  for  their  feedback,  participants  in  the  data 
collection  indicated  that,  after  going  through  the  process, 
they  had  an  overall  positive  opinion  about  the  JANUS 
technique  and  thought  that  the  questions  asked  were 
relevant  to  OE  causal  factors.  In  a  separate  activity,  FAA 
managers  and  NATCA  facility  representatives  from  the 
participating  facilities  indicated  that  the  method  could 
be  a  useful  tool  if  implemented  operationally.  A  separate 
group  of  ATC  management  and  staff  personnel  who  were 
not  associated  with  the  beta  test  rated  the  data  output  from 
JANUS  as  more  comprehensive,  informative,  practical, 
specific,  and  useful  compared  with  the  output  from  the 
current  process  used  to  identify  OE  causal  factors  (as 
described  in  FAA  Order  7210.56). 

To  assess  information  value  of  the  JANUS  technique,  79 
OEs  (64  from  ARTCCS  and  1 5  from  terminals)  obtained 
during  the  beta  test  (Pounds  &  Isaac,  2003)  were  used 
to  compare  the  causal  factors  identified  by  JANUS  with 
those  identified  by  the  current  OE  investigation  process. 
One  difference  between  the  techniques  is  that  the  cur¬ 
rent  FAA  OE  investigation  process  analyzes  the  OE  as 
a  unitary  event  whereas  the  JANUS  technique  permits 
splitting  each  OE  into  several  “links  in  the  chain”  and 
analyzing  each  link  separately. 

In  the  sample  of  79  OEs,  133  causal  factor  items  were 
reported  (an  average  of  1.7  causal  factors  per  OE)  and 
were  distributed  in  the  following  categories  found  on  the 
current  OE  reporting  form:  Data  Posting  (9.8%),  Radar 
Display  (58.7%),  Aircraft  Observation  (Towers  Only; 
1.5%),  Communication  Error  (25.6%),  Coordination 
(4.5%),  and  Position  Relief  Briefing  (0%).  In  this  same 
sample ,  an  average  of  3 . 6  factors  per  OE  was  related  to  the 
mental  processes  of  the  controller  working  the  traffic.  The 
factors  were  distributed  in  the  following  categories:  Per¬ 
ception  and  Vigilance  (41%),  Memory  (15%),  Planning 
and  Decision  Making  (49%),  and  Response  Execution 
(10%).  (Because  the  JANUS  technique  permitted  more 


than  one  category  to  be  selected,  the  percentage  summed 
across  categories  was  greater  than  100%.)  Contextual 
factors  that  influenced  the  OEs  were  also  identified  for 
each  “link”  resulting  in  an  average  of  9.6  factors  per  OE. 
The  factors  were  distributed  in  the  following  categories: 
Traffic  &  Airspace  (49%),  Weather  (28%),  Teamwork 
(26%),  Pilot  Actions  (21%),  Individual/Personal  Factors 
(2 1  %) ,  Pilot-Controller  Communications  (20%) ,  Ambi¬ 
ent  Environment/Distractions  (18%),  Workplace/Equip- 
ment/Ffuman-Machine  Interactions  (13%),  Procedures 
and  Orders  (11%),  Training  and  Experience  (10%), 
Supervision  and  Management  (10%),  Organizational 
Factors  (10%),  Interpersonal/Social  Factors  (5%),  and 
Documents  and  Materials  (0.3%). 

Experiments 

None  of  the  experiments  identified  in  the  literature  search 
studied  OEs  as  the  criterion  variable  of  interest.  Although 
it  is  possible  to  create  an  experimental  manipulation  in 
which  the  likelihood  of  committing  an  OE  is  high  (but 
would  not  necessarily  be  guaranteed  to  occur),  none  of 
the  experiments  was  designed  in  that  manner.  Instead, 
the  focus  was  on  assessing  the  impact  of  experimental 
manipulations  on  typical  OE  predictors,  especially  those 
related  to  SA,  memory,  and  workload.  The  most  com¬ 
mon  experimental  manipulation  involved  varying  traffic 
volume  to  assess  aspects  of  controller  performance. 

The  primary  hurdle  involved  in  using  simulations  in 
experiments  to  assess  OE  occurrence  is  the  infrequency 
with  which  OEs  occur.  Given  the  vast  numbers  of  com¬ 
munications  and  control  actions  issued  correctly  during 
the  course  of  a  single  day,  we  would  expect  to  see  few  OEs 
occur  during  the  course  of  a  two-hour  (or  even  longer) 
simulation.  When  we  conduct  an  experiment  to  assess  the 
effects  of  the  amount  of  traffic  on  controller  performance, 
we  can  clearly  measure  how  it  might  influence  SA  and 
radar  scanning.  However,  are  these  performance  changes 
predictive  of  OE  occurrence?  For  example,  suppose  an 
SA  experiment  reveals  that  controllers’  radar  scanning 
and  SA  are  degraded  under  conditions  of  high  aircraft 
volume.  Can  we  then  extrapolate  to  say  that  reductions 
in  the  amount  of  traffic,  which  resulted  in  improved  radar 
scanning  and  SA,  are  likely  to  reduce  OEs?  This  conclu¬ 
sion  would  likely  be  invalid,  since  we  already  know  that 
a  significant  number  of  OEs  occur  under  lower  traffic 
loads.  Thus,  organizational  interventions  designed  to 
improve  SA  may  be  very  successful  at  improving  SA  but 
appear  to  be  ineffective  in  reducing  OEs.  Until  we  can 
identify  aspects  of  controller  performance  that  are  predic¬ 
tive  of  OE  occurrence,  it  is  highly  unlikely  that  we  can 
utilize  simulations  and  experimental  methods  to  define 
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effective  OE  intervention  strategies.  Some  would  argue 
that,  given  the  dynamic  nature  of  the  air  traffic  environ¬ 
ment,  we  are  not  likely  to  identify  discrete  performance 
predictors  of  OEs. 

We  do  not  argue  that  conducting  experiments  is  unim¬ 
portant.  On  the  contrary,  continued  simulation-based 
studies  are  needed  to  improve  our  understanding  of 
the  relationships  among  OE  predictor  variables.  As  we 
enhance  our  understanding  of  the  role  of  experience, 
techniques,  and  amount  of  traffic  on  controller  perfor¬ 
mance,  we  will  gain  a  better  understanding  of  predictors 
and  prevention  methods  for  OEs. 


tion  of  OEs,  these  percentages  instead  reflect  the  general 
interest  of  the  researchers,  as  well  as  the  availability  of 
information  about  the  topic  areas. 

While  a  short  description  of  the  literature  in  each  cat¬ 
egory  will  be  provided,  the  high  percentage  of  documents 
classified  as  belonging  to  the  Contextual  Conditions  and 
Mental  Processes  categories  allows  more  discussion  about 
the  literature  in  those  categories.  For  a  complete  list  of 
reports  classified  in  each  of  the  JANUS  categories,  refer 
to  Appendix  B. 

Organizational  and  Management  Factors 


Classifying  the  Literature  by  JANUS  Categories 

Figure  4  shows  the  number  of  reports  that  fell  within 
each  of  the  high-level  JANUS  HE  categories.  Because  a 
document  could  address  several  JANUS  categories,  the 
total  number  of  citations  included  in  Figure  4  exceeds 
the  number  of  source  documents. 

The  OE  literature  fell  primarily  into  two  of  the  high-level 
JANUS  categories:  contextual  conditions  associated  with 
an  OE  (44%  of  the  reports)  and  mental  processing  of 
involved  controllers  (30%  of  the  reports).  The  literature 
also  addressed:  organizational  influences  (7%),  manage¬ 
rial  influences  (4%),  supervisory  influences  (6%),  con¬ 
troller  non-compliance  (1%),  the  controller  task  (1%), 
and  controller’s  response  execution  to  choice  and  plans 
(7%).  Rather  than  serving  as  an  indication  of  the  relative 
importance  of  a  given  JANUS  category  in  the  investiga¬ 


Using  a  structured  interview  process  based  on  the  EtFACS 
aircraft  accident  classification  system  (Shappell  and  Wieg- 
mann,  2000b),  the  National  Aviation  Research  Institute 
(Conner  &  Corker,  2001)  interviewed  109  controllers 
at  AT  facilities  that  had  high  or  low  error  rates.  They 
found  that,  while  interviewees  rated  controller  perfor¬ 
mance  as  the  most  important  factor  in  error  causation, 
organizational  factors  were  also  viewed  as  being  of  high 
importance.  Their  list  of  most  important  factors  also 
included  organizational  climate. 

In  an  unpublished  work  group  report  about  the  causes  of 
surface  incidents,  Kauliaet  al.  (1987)  stated  that,  in  addi¬ 
tion  to  factors  related  to  controllers,  various  organizational 
factors  needed  to  be  addressed.  These  included:  a)  the 
lack  of  updated  Standard  Operating  Procedures  (SOPs), 
b)  an  insufficient  number  of  trained  supervisors,  and  c) 
supervisors  overburdened  with  administrative  duties. 
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Figure  4.  Distribution  of  OE-related  documents  by  JANUS  category. 
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Supervision 

Going  back  as  far  as  Kinney  (1977b,  1977c),  supervi¬ 
sory  influences  on  ATC  OEs  were  noted  and  echoed  by 
various  ATC  work  group  reports  that  emerged  in  the 
1980s  following  the  PATCO  strike.  Kaulia  et  al.  (1987) 
identifled  these  supervisory  concerns:  The  inability  to 
a)  correct  the  actions  of  unsafe  controllers,  b)  monitor 
controller  performance,  and  c)  ensure  that  controllers 
were  properly  trained. 

Another  aspect  of  supervision  that  has  received  limited 
research  attention,  yet  appears  to  have  some  potential 
for  being  a  factor  in  reducing  OEs,  involves  aspects 
of  the  supervisor’s  duties  and  responsibilities.  As  part 
of  a  survey  of  supervisors,  Kirk,  Mayberry,  and  Lesko 
(1996),  asked  for  an  indication  of  their  time  allocation 
across  eight  functional  areas  (supervisor  operations,  man¬ 
age  resources,  monitor  performance,  provide  training, 
maintain  expertise,  foster  teamwork,  collateral  duties, 
and  personnel  issues)  and  then  asked  them  to  rate  the 
importance  of  each  of  the  above  functions  on  a  5-point 
scale.  Supervisor  time  was  primarily  allocated  to  one  of 
four  functions:  enhancing  operational  effectiveness,  man¬ 
aging  resources,  interpersonal  skills,  and  miscellaneous 
assignments.  Supervising  the  operation  was  viewed  as  the 
most  important  function  across  all  facilities. 

For  en  route  centers,  Kirk  et  al.  examined  the  relation¬ 
ship  between  the  importance  ratings  provided  by  the 
supervisors  and  OEs.  Team  supervisors  who  placed  greater 
emphasis  on  the  importance  of  managing  resources  and 
supervising  the  operation  had  fewer  OEs.  The  emphasis 
the  area  supervisor  in  charge  (ASIC)  placed  on  several 
other  functions  (promotion  of  teamwork,  monitor  perfor¬ 
mance,  provide  training,  and  address  personnel  issues)  was 
also  significantly  related  to  fewer  OEs.  The  implication 
is  that  ASICs  who  rated  the  importance  of  these  duties 
and  responsibilities  highly  were  more  likely  to  have  fewer 
OEs  in  their  role  as  an  ASIC.  For  example,  75%  of  the 
ASICs  who  said  that  managing  resources  was  among  the 
most  important  supervisory  function  had  one  OF  or  less 
in  their  role  as  a  supervisor.  Those  who  viewed  managing 
resources  as  the  least  important  supervisory  function  had 
a  median  number  of  two  OEs.  There  was  no  evidence 
of  a  significant  relationship  between  allocation  of  time 
between  the  supervisory  functions  and  the  occurrence 
of  OEs.  Additional  research  is  needed  to  understand 
how  these  attitudes  are  translated  into  behaviors  in  the 
workplace.  Using  the  OF  database,  Kirk,  Mayberry,  and 
Lesko  (1996)  suggested  that  there  was  no  direct  relation¬ 
ship  between  supervisory  activity  at  the  time  of  the  error 
and  the  error’s  severity. 


Kirk  et  al.  ( 1 996)  also  examined  other  aspects  of  the  effects 
of  supervisory,  managerial,  and  organizational  influences 
on  OEs  at  en  route  centers.  They  based  their  assessment 
on:  a)  visits  to  selected  en  route  centers,  b)  interviews 
with  controllers,  managers,  and  supervisors,  and  c)  re¬ 
views  of  previous  studies.  Their  extensive  investigation 
of  the  role  of  supervisors  involved  several  dimensions 
including  supervisory  duties,  training  and  development, 
staffing  changes,  teamwork,  supervisors’  background,  and 
empowerment.  Their  assessment  of  changes  in  staffing 
took  into  account  the  potential  influence  of  a  govern¬ 
ment-wide  effort  (NPR)  to  increase  the  span  of  control 
for  supervisors.  Within  the  FAA,  this  meant  that  efforts 
were  focused  on  increasing  the  employee/supervisor  ra¬ 
tios  from  around  7:1  to  the  proposed  government- wide 
target  of  15:1.  Subsequent  reorganization  efforts  at  the 
larger  en  route  facilities  led  to  a  significant  reduction  in 
first-line  supervisor  staffing.  For  example,  from  1992  to 
1995,  supervisory  staffing  levels  declined  from  around 
1 1%  at  ZTL  to  nearly  40%  at  ZLC.  Over  this  same  time 
period,  there  was  a  gradual  increase  in  the  number  of  OEs. 
Kirk,  Mayberry,  &  Lesko  (1996)  found  that  there  was  a 
statistically  significant  relationship  between  the  decline 
in  supervisory  staffing  and  the  overall  increase  in  OEs 
(explaining  14%  of  the  overall  variance).  The  relation¬ 
ship  was  slightly  stronger  when  they  took  into  account 
the  ratio  of  supervisors  to  controllers  in  each  of  the  en 
route  centers.  A  similar  relationship  was  not  found  for 
theTRACON  or  Tower  comparisons.  As  a  result  of  their 
observations,  interviews,  and  analyses,  the  authors  made 
the  following  recommendations. 

1 .  Develop  a  standardized  way  of  measuring  sector  com¬ 
plexity  so  that  sector  staffing  can  be  adjusted  based 
on  this  metric  instead  of  the  amount  of  traffic. 

2.  Develop  action  plans  for  high-OE  sectors,  which 
included  the  need  for  more  managerial  attention  to 
be  directed  to  these  sectors  and  an  increase  in  the 
level  of  supervision. 

3.  Monitor  the  performance  of  less  experienced  con¬ 
trollers  more  closely. 

4.  Match  controller  work  assignments  to  the  amount 
of  expertise  required. 

5.  Identify  the  supervisor-controller  factors  that  affect 
controller  performance  during  an  OE. 

6.  Minimize  the  amount  of  managerial  turnover  to 
stabilize  the  culture. 

7.  Improve  the  evaluation  of  controller  training  and 
performance. 

8.  Focus  more  attention  on  team  training. 

NARI  (Conner  &  Corker,  2001)  found  that  controllers 
tended  to  view  supervisory  practices  as  a  significant  factor 
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Contextual  Conditions 


Figure  5.  Distribution  of  OE-related  reports  by  sub-categories  of  JANUS  Contextual  Conditions. 


that  both  contribute  to  OEs  and  serve  as  a  positive  force 
to  avoid  OEs.  Broach  and  Dollar  (2002)  provided  an 
expanded  look  at  the  role  of  the  supervisor-to-controller 
ratio  (SCR)  in  OEs  by  also  taking  into  account  several 
organizational  factors  derived  from  the  1997  and  2000 
FAA  employee  attitude  surveys.  Their  analysis  focused 
on  en  route  OEs  that  occurred  between  1997  and  2000. 
They  reported  that  two  organizational  factors  (employee 
perceptions  of  equipment/facilities  and  performance 
management)  along  with  the  SCR  accounted  for  50%  of 
the  overall  variance  in  OE  rates.  Outcomes  from  Broach 
and  Dollar  (2002),  Kirk,  Mayberry,  and  Lesko  (1996), 
and  other  studies  clearly  demonstrate  the  importance 
of  a  number  of  supervisory/  managerial/organizational 
factors  in  OEs.  Additional  efforts  are  needed  to  further 
identify  and  clarify  the  manner  in  which  these  factors 
influence  OE  occurrence. 

Contextual  Conditions 

Forty- four  percent  of  the  OE-related  documents  fell  under 
the  JANUS  Contextual  Conditions  category.  Figure  5 
shows  the  distribution  of  documents  classifled  into  the 
Contextual  Conditions  sub-categories.  Five  sub-categories 
accounted  for  66%  of  the  Contextual  Conditions:  a)  the 
combination  of  the  Traffic  and  Airspace^  sub-categories 
(15%),  b)  Pilot-ATC  Communications  (10%),  c)  Team¬ 
work  (12%),  d)  Training  and  Experience,  (14%),  and 


^  While  discussed  separately,  we  have  combined  the  Amount  of 
Traffic  and  Airspace  sub-categories  here  because  they  are  interactive 
in  their  effects. 


e)  Human  Machine  Interface  (HMI)  and  Equipment 
(15%).  The  literature  from  each  of  these  sub-categories 
will  be  discussed  in  some  detail.  For  a  listing  of  the 
reports  classified  as  belonging  to  the  remaining  JANUS 
contextual  conditions  subcategories,  the  reader  is  referred 
to  Appendix  B. 

Traffic/Airspace.  Two  contextual  factors  under  traffic/ air¬ 
space  that  have  received  the  greatest  attention  since  the 
1960s  are  the  amount  of  traffic  and  airspace  complexity. 
Traffic  is  a  dynamic  factor,  while  airspace  (including  air¬ 
port  surface  characteristics)  is  static,  so  these  two  factors 
will  be  discussed  separately. 

1)  Amount  ofTraffic.  Kershner  (1968)  conducted  an  initial 
study  on  the  amount  of  traffic  and  its  relationship  with 
age,  workload,  and  time-on-shift  in  1965-66  OEs.  Spahn 
(1977)  looked  at  changes  in  air  traffic  levels  from  1974 
to  1976.  Spahn  pointed  out  that  a  higher  percentage  of 
OEs  occurred  under  light  and  moderate  traffic  rather  than 
during  heavy  traffic.  From  the  1970s  through  the  early 
1980s,  the  trend  was  toward  an  increased  percentage  of 
OEs  under  light  and  moderate  versus  heavy  traffic.  Us¬ 
ing  information  from  previous  studies,  Schroeder  (1982) 
provided  information  regarding  changes  in  the  percent¬ 
age  of  OEs  occurring  under  the  three  traffic  levels  (see 
Figure  6).  From  1965  through  1980,  there  was  a  gradual 
increase  in  OEs  occurring  under  light  traffic  levels  with 
a  corresponding  decline  in  the  percentage  occurring 
under  heavy  traffic  levels.  However,  the  categorization  of 
air  traffic  required  the  investigator  to  make  a  subjective 
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Amount  of  Traffic 


Light  Moderate  Heavy 

Figure  6.  Distribution  of  amount  of  traffic  associated  with  OEs  by  year. 


rating.  There  was  little  assurance  that  investigators  within 
and  across  facilities  used  the  same  criteria  for  judging  the 
amount  of  traffic  at  the  time  of  the  OE. 

As  part  of  a  MITRE  study  to  assess  the  validity  of  com¬ 
mon  statements  regarding  the  relationship  between  traffic 
and  OEs,  Lowry,  Mac  Williams,  Still,  &  Walker  (2005) 
conducted  several  analyses  of  the  NAIMS  1 999-2003  OE 
database.  The  percentage  of  OEs  that  involved  only  two 
aircraft  was  small  (1%  in  en  route,  4%  in  TRACONs, 
and  7%  in  towers) .  In  en  route  facilities,  2 1  %  of  the  OEs 
invoIved7-8  aircraft.  Our  analysis  ofthe  database  suggests 
that  this  percentage  has  remained  relatively  consistent 
across  time.  While  not  discussed  in  their  report,  the 
figures  used  to  relate  OEs  to  traffic  suggest  that  slightly 
more  than  50%  of  the  OEs  in  TRACONS  and  towers 
involved  between  4-6  aircraft. 

A  second  question  addressed  by  Lowry  et  al.  (2005) 
was  whether  OEs  occur  just  prior  to  the  traffic  peak.  To 
answer  this  question,  the  authors  analyzed  52  OEs  that 
occurred  in  FY02  and  determined  the  traffic  count  30 
minutes  prior  to  and  30  minutes  following  the  OE.  In 
6 1  %  of  the  OEs,  traffic  had  already  peaked  when  the  OE 
occurred,  in  another  30%  of  the  cases  the  OE  occurred 
prior  to  the  peak  in  traffic. 


As  evident  in  reports  that  described  aspects  of  the  1986 
and  1987  OEs  (FAA,  1988),  trends  in  OE  occurrence 
associated  with  the  day  of  the  week  and  time  of  day  ap¬ 
peared  to  be  related  to  overall  changes  in  the  amount  of 
traffic.  OEs  tended  to  peak  on  Wednesdays  and  Thurs¬ 
days,  with  the  lowest  OE  rates  evident  on  Saturdays  and 
Sundays.  Throughout  the  day,  OEs  increased  with  a  peak 
around  9  to  10  a.m.,  leveled  off,  followed  by  a  second¬ 
ary  and  somewhat  higher  peak  around  4  to  6  pm.  With 
respect  to  the  controller’s  8-hour  workday,  the  lowest 
number  of  OEs  occurred  during  the  seventh  and  eighth 
hour  of  the  shift.  These  trends  are  generally  consistent 
across  years  and  are  evident  in  the  current  OE  database 
(Eowry  et  ah,  2005). 

2)  Airspace  complexity.  Besides  considering  the  role  of  the 
amount  of  traffic  in  OEs,  Spahn  ( 1 977)  also  looked  at  the 
influence  of  traffic  complexity.  At  the  time,  information 
included  in  the  OE  database  about  both  the  amount  of 
traffic  and  traffic  complexity  required  subjective  judg¬ 
ments  to  be  made  by  the  investigator.  These  two  measures, 
the  amount  of  traffic  and  traffic  complexity,  are  not 
independent.  As  the  amount  of  traffic  increases,  there 
is  generally  a  corresponding  increase  in  traffic  complex¬ 
ity.  Spahn’s  (1977)  data  suggest  an  interaction  between 
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traffic  complexity  and  the  amount  of  traffic,  with  higher 
percentages  of  OEs  occurring  in  moderate  and  heavy 
traffic  complexity  conditions.  This  suggests  that  the 
interaction  of  amount  of  traffic  with  traffic  complexity 
may  affect  OEs  more  than  the  amount  of  traffic  alone. 
However,  there  are  significant  questions  concerning  the 
definition  and  measurement  of  both  traffic  and  airspace 
complexity. 

Most  of  the  research  on  airspace  complexity  has  been 
targeted  at  developing  measures  of  dynamic  density, 
an  indicator  of  future  sector  activity,  rather  than  OEs. 
However,  several  reports  have  examined  the  relationship 
between  sector  complexity  and  OEs  at  en  route  facilities. 
For  example,  Grossberg  (1989)  found  a  correlation  of  .44 
between  an  index  of  sector  complexity  based  on  the  top 
four  complexity  factors  (complex  control  adjustments, 
climbing  and  descending  flight  paths,  large  airspace,  and 
mix  of  aircraft  types)  and  the  number  of  operational  errors 
at  sectors  in  the  Chicago  Center.  Rodgers,  Mogford,  and 
Mogford  (1998)  related  Mogford  s  (Mogford  et.  al.,  1994) 
1 6  complexity  factors  to  the  incidence  of  operational  errors 
at  Atlanta  Center  (ZTL)  sectors.  They  found  that  a  vari¬ 
able  that  characterized  sectors  as  having  a  low,  medium, 
or  high  incidence  of  OEs  predicted  several  of  the  sector 
complexity  factors.  They  also  found  that  several  sector 
complexity  factors  (notably  radio  frequency  congestion 
and  special  use  airspace  activity)  predicted  the  number 
of  errors  per  sector  in  an  analysis  of  ZTL  data.  Thus, 
they  concluded  that  sector  characteristics  appeared  to 
be  related  to  OEs. 

NARl  (Conner  &  Corker,  2001)  defined  Operational 
Complexity  as  “the  characteristics  of  air  traffic  control 
processes,  environment,  and  the  users  who  operate  in 
it,  including  the  airspace,  equipment,  traffic,  weather, 
aircraft  performance,  military,  information,  frequencies, 
etc.”  Interviews  conducted  during  the  NARI  OE  study 
(2001)  identified  Operational  Complexity  as  the  second 
highest-rated  OE  causal  factor.  Moreover,  Pounds  and 
Isaac  (2003)  found  that  49%  of  the  errors  identified  in 
their  JANUS  validation  study  were  related  to  either  sector 
or  traffic  characteristics. 

Lowry  et  al.  (2005)  conducted  several  analyses  of  the 
1 999-2003  database  that  focused  on  issues  often  included 
in  assessments  of  airspace  complexity  (sector  size,  sector 
configuration,  and  traffic  mix).  With  respect  to  sector 
size  at  en  route  centers,  their  analysis  supported  previous 
research  by  Rodgers  et  al.  (1998)  indicating  that  smaller 
sectors,  with  a  volume  of  50,000  or  less  volume  units, 
comprised  50%  of  the  sectors,  handled  52%  of  the  traffic, 
and  experienced  64%  of  the  OEs.  With  respect  to  traffic 


mix,  neither  regional  jets  nor  business  jets  were  involved 
in  a  disproportionate  number  of  OEs.  However,  in  ter¬ 
minal  facilities  the  authors  found  that  turboprops  were 
disproportionately  involved  in  OEs  in  three  of  the  four 
facilities  included  in  the  analysis.  An  additional  analysis 
was  focused  on  the  impact  of  a  National  Choke  Point 
Initiative  in  which  two  sectors  at  the  ZID  and  ZOB  en 
route  centers  were  redesigned  to  improve  efficiency.  While 
Massimini,  Nene,  Gormley,  &  Stevens,  (2001)  found 
that  the  redesign  reduced  demand  on  adjacent  sectors 
and  improved  efficiency,  Lowry  et  al.  (2005)  found  that 
the  OE  rates  two  years  prior  to  and  two  years  following 
the  redesign  were  the  same. 

Pilot-ATC  Communications.  Most  empirical  ATC  studies 
have  been  conducted  to  assess  the  phraseology  associated 
with  Pilot-ATC  communications.  Initial  studies  involving 
analyses  of  the  FAA/NASA  Aviation  Safety  Reporting 
System  (ASRS)  database  were  conducted  about  20  years 
ago  (Grayson  &  Billings,  1981;  Monan,  1980;  Monan, 
1983).  Initial  efforts  were  focused  on  factors  associated 
with  miscommunications  (Grayson  &  Billings,  1981). 
They  included  pilot  expectations,  similar  callsigns,  trans¬ 
posing  numbers,  garbled  phraseology,  and  inaccurate  or 
incomplete  messages.  Monan  (1980,  1983)  investigated 
the  role  of  similar  call  signs  as  well  as  readback/hearback 
errors.  Cardosi,  Falzarano,  and  Han  (1998)  conducted 
a  more  recent  review  of  the  communications  error  in¬ 
formation  contained  in  the  ASRS  reports  as  a  means  of 
identifying  factors  that  contributed  to  the  errors.  The 
most  common  factors  involved  similar  call  signs,  pilot 
expectations,  and  high  controller  workload.  While  not 
focused  solely  on  OEs,  these  studies  provide  an  overall 
perspective  for  understanding  some  of  the  communica¬ 
tions  issues  associated  with  OEs. 

Cardosi  and  her  colleagues  performed  much  of  the  re¬ 
search  focused  on  the  role  of  voice  communications  as 
a  critical  safety  link  in  the  NAS.  A  recent  report  on  the 
metrics  of  communication  performance  by  Cardosi  and 
DiFiore  (2004)  provides  an  excellent  summary  of  their 
work,  some  of  which  is  now  a  decade  old  and,  thus,  may 
represent  a  conservative  estimate  of  communications  in 
today’s  environments.  The  metrics  incorporated  in  their 
studies  included:  (i)  number  of  controller  transmissions 
per  minute,  (ii)  number  of  clearances  issued  per  minute, 
(iii)  characteristics  of  pilot  responses  to  clearances,  (iv) 
percentage  of  clearances  that  result  in  a  pilot  readback 
error,  (v)  percentage  of  readback  errors  that  are  not 
corrected  by  the  controller,  (vi)  percentage  of  control¬ 
ler  transmissions  that  need  to  be  repeated  due  to  pilot 
requests,  (vii)  factors  associated  with  miscommunications, 
and  (viii)  the  time  required  to  successfully  transmit  an 
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instruction  to  maneuver  for  traffic  avoidance  in  the  en 
route  environment.  Each  of  the  studies  involved  an 
analysis  of  approximately  50  hours  of  voice  tapes  from 
tower  cabs,  TRACON,  and  en  route  air  traffic  control 
facilities  (Cardosi,  1993;  Burki-Cohen,  1995;  Cardosi, 
1994;  and  Cardosi,  Brett,  &  Han,  1996). 

We  have  elected  to  summarize  the  overall  results  described 
in  Cardosi  and  DiFiore  (2004)  rather  than  describe  more 
detailed  outcomes  from  each  of  the  studies.  Evidence 
suggests  that  the  number  of  controller  transmissions 
varies  by  environment,  from  a  low  of  just  under  2  to  4.5 
communications  per  minute  in  TRACONs  and  about 
8  communications  per  minute  for  the  tower  Ground 
Control  position.  Of  course  the  average  number  of  com¬ 
munications  for  Ground  Control  may  vary  considerably 
from  larger  to  smaller  towers.  Pilot  responses  to  controller 
communications  varied  considerably;  at  en  route  centers 
some  70%  contained  a  full  readback,  while  at  the  tower 
Local  Control  position,  less  than  30%  involved  a  full 
readback. 

A  consistent  finding  in  Cardosi’s  work  was  that  the  rate 
of  communication  errors,  defined  as  readback  errors  and 
requests  for  repeated  communications,  was  low  across 
facility  types.  The  readback  error  rate  was  consistently  less 
than  1%;  requests  for  repeated  transmissions  was  similarly 
low,  with  the  exception  of  the  en  route  environment, 
where  the  frequency  was  slightly  higher,  at  1.4%.  While 
a  readback  error  occurred  once  every  2.5  hours  for  local 
controllers,  at  TRACONs  the  rate  was  around  one  every 
half-hour.  The  vast  majority  of  these  errors  were  caught 
and  corrected  by  the  involved  controllers. 

Causal  factors  associated  with  communication  errors 
(few  of  which  lead  to  OEs)  included  attempting  to 
convey  too  much  information,  frequency  congestion, 
workload,  speech  rate,  poor  grasp  of  the  English  language 
by  international  pilots,  and  similar-sounding  aircraft  call 
signs.  Analyses  of  the  data  by  Cardosi  and  her  colleagues 
suggest  that  there  is  a  strong  relationship  between  the 
complexity  of  the  controller’s  transmission  and  the  prob¬ 
ability  of  a  readback  error  in  the  en  route  and  TRACON 
environments.  For  en  route  transmissions,  the  readback 
error  rate  doubled  as  complexity  increased  from  three 
to  four  elements.  While  4%  of  controller  transmissions 
involved  five  or  more  elements,  they  contributed  26% 
of  the  errors  in  en  route  facilities.  A  similar  finding  was 
observed  in  TRACONs. 

Although  nearly  all  communications  are  free  of  errors, 
communication  errors  play  a  significant  role  in  the  overall 
OE  process.  Rodgers  and  Nye  (1993)  found  that  36% 


of  en  route  OEs  involved  communication  errors.  In  the 
tower  environment,  Cardosi  and  Yost  (2001)  found  that 
41%  of  OEs  involved  communication  errors,  with  19% 
involving  pilot-controller  miscommunications. 

Baseline  field  studies  such  as  those  conducted  by  Cardosi 
provide  a  necessary  perspective  on  the  infrequency  of 
communication  errors  in  the  course  of  normal  operations. 
For  example,  imagine  an  intervention  designed  to  reduce 
hearback/ readback  errors.  Usingapre- vs.  post-interven¬ 
tion  evaluation  design,  we  could  determine  the  effective¬ 
ness  of  the  intervention  at  reducing  hearback/ readback 
errors.  The  effectiveness  of  the  intervention  is  not  based 
on  the  reduction  of  hearback/ readback  errors  that  occur 
during  OEs.  Rather,  the  intervention  is  being  evaluated 
against  hearback/ readback  errors  that  occur  during  all 
communication  exchanges.  Given  that  less  than  0.5% 
of  all  communication  exchanges  between  controllers  and 
pilots  result  in  uncorrected  hearback/ readback  errors,  and 
that  only  a  fraction  of  this  0.5%  of  communications  will 
actually  involve  an  OE,  it  would  be  difficult  in  a  field 
study  to  evaluate  the  effectiveness  of  such  an  intervention 
on  the  reduction  of  OEs,  per  se.  This  is  an  important 
factor  to  keep  in  mind  as  we  review  the  literature.  Causal 
factors  commonly  associated  with  OEs  (i.e.,  data  post¬ 
ing  errors,  radar  display  errors,  tower  observation  errors, 
communication  errors,  coordination  errors,  and  position 
relief  briefing  errors)  occur  at  some  base  rate  within  nor¬ 
mal  operations.  Even  if  an  intervention  reduces  the  base 
rate  with  which  these  errors  occur,  the  extent  to  which 
the  intervention  will  have  a  significant  impact  on  OEs 
remains  to  be  seen. 

The  safe  and  efficient  flow  of  traffic  in  the  NAS  relies 
heavily  on  voice  communications  between  pilots  and 
controllers.  As  evident  from  the  results  presented  earlier, 
OEs  are  frequently  associated  with  readback  errors.  In  their 
summary  report,  Cardosi  and  DiFiore  (2004)  indicated 
that  communication  errors  appear  to  be  closely  linked 
with  the  complexity  of  the  controller’s  transmission.  While 
this  research  has  been  focused  on  the  controller  side  of  the 
communication  link,  less  information  is  available  about 
factors  that  contribute  to  a  pilot’s  failure  to  understand 
and  respond  properly  to  a  communication.  The  excep¬ 
tion  involves  research  focused  on  age  differences  in  pilot 
responses  to  air  traffic  control  instructions.  While  these 
studies  were  not  focused  on  OEs,  they  provide  insight 
into  factors  that  can  lead  to  miscommunications  and 
possible  OEs. 

Since  the  complexity  of  a  controller’s  transmission  is 
linked  with  communication  failures,  we  can  assume  that 
working  memory  capacity  plays  an  important  role  in 
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pilot  controller  communications.  Research  has  demon¬ 
strated  an  age-related  decline  in  working  memory  capac¬ 
ity  (Baddeley  &  Hitch,  1974)  and  speed  of  processing 
(Salthouse,  1980)  as  well  as  an  increased  susceptibility 
to  interference  or  distraction  (Engle,  Kane,  &  Tuholski, 
1999).  Research  has  focused  on  determining  the  extent  to 
which  pilot  expertise  may  reduce  the  typical  age-related 
decline  in  the  recall  of  ATC  communications.  Previous 
research  demonstrated  that  older  pilots  exhibited  a  slightly 
lower  ability  to  recall  ATC  communications  (Morrow, 
Leirer,  &  Yesavage,  1990;  Morrow,  Yesavage,  Leirer,  & 
Tinklenberg,  1993;  Taylor  et  al.,  1994;  Yesavage  et  al., 
1999).  However,  Morrow  &  Leirer  (1997)  and  Morrow, 
Leirer,  Altieri,  &  Pitzsimmons  (1994)  found  that  pilot 
expertise  may  moderate  age  differences  in  recalling  ATC 
communications. 

In  a  recent  study  (Taylor  et  al.,  2005)  the  authors  set  out 
to  assess  the  role  of  cognitive  ability  and  domain-specific 
expertise  as  they  influence  age-related  differences  in  pilot 
responses  to  ATC  messages.  In  addition  to  assessing  the 
role  of  pilot  expertise,  as  determined  by  pilot  ratings  (VPR, 
IPR,  and  CPII/ATP),  the  authors  varied  the  length  and 
speech  rate  of  the  ATC  instructions  for  pilots  who  were 
not  allowed  to  write  down  the  communications  while 
flying  a  simulator.  The  97  pilots  in  the  study  were  split 
into  two  groups  for  the  age  comparisons  (45  to  57  and  57 
to  69).  The  results  demonstrated  that  pilots  recalled  2.05 
of  3  instructions  when  the  instructions  were  presented  at 
a  normal  speech  rate,  but  performance  declined  to  1.88 
instructions  recalled  for  the  fast  speech  rate.  Also,  when 
3  instructions  were  presented,  an  average  of  2.05  was 
recalled,  but  longer  ATC  messages  led  to  lower  accuracy 
(only  an  average  of  1.87  instructions  was  recalled).  It 
would  seem  likely  that  pilots  would  recall  ATC  com¬ 
munications  better  if  they  were  allowed  to  write  down 
elements  of  the  communications  as  is  commonly  done 
while  flying. 

While  this  study  showed  that  expertise  led  to  improved 
recall  of  ATC  communications,  there  was  little  evidence 
that  it  reduced  the  age-related  decline  in  performance. 
They  found  that  the  effect  of  age  on  recall  accuracy  was 
largely  explained  as  an  “age-associated  decrease  in  working 
memory  span,  which  in  turn  was  explainable  as  decreases 
in  both  speed  and  interference  control  (pg.  1 17).”  These 
results,  along  with  those  of  Cardosi  and  her  colleagues, 
can  be  used  to  support  interventions  designed  to  improve 
communications.  The  emphasis  of  these  interventions 
should  be  on  a  standard  speech  rate  and  shorter  instruc¬ 
tional  messages.  The  results  also  emphasize  the  importance 
of  involving  pilots  in  efforts  to  reduce  hearback/readback 
and  other  communication  failures. 


Teamwork.  ATC  teamwork  is  generally  discussed  in  the 
literature  relative  to  either  (a)  controller  task  coordina¬ 
tion,  or  (b)  interpersonal  skills.  Discussions  of  task 
coordination  primarily  involve  controller  team  members 
functioning  within  dyads,  such  as  a  radar  controller  and 
radar  associate,  a  ground  and  local  controller,  or  two 
controllers  involved  in  inter-sector  coordination.  One 
of  the  more  visible  aspects  of  task  coordination  occurs 
during  position  changes. 

Position  Relief  Briefings.  One  aspect  of  teamwork  is  the 
transfer  of  position  responsibility  that  takes  place  when 
one  controller  relieves  a  second  controller.  This  process 
is  referred  to  as  a  position  relief  briefing.  The  current 
controller  handbook  (PAA,  2006)  contains  rather  specific 
instructions  regarding  the  process  for  accomplishing 
a  position  relief  briefing.  During  this  time,  a  relieving 
controller  observes  the  traffic  situation  that  he/ she  will  be 
taking  over.  The  controller  being  relieved  then  follows  a 
checklist  to  ensure  that  all  relevant  information  has  been 
transferred  to  the  relieving  controller. 

The  current  NATCA  contract  (PAA,  1998a)  states,  “Un¬ 
less  operational  requirements  do  not  permit,  employees 
shall  not  be  required  to  spend  more  than  two  (2)  con¬ 
secutive  hours  performing  operational  duties  without 
a  break  away  from  operational  areas.”  This  means  that 
during  the  course  of  a  24-hour  day,  several  position  relief 
briefings  are  conducted  as  one  controller  leaves  a  position 
and  another  takes  over. 

The  identification  of  position  relief  briefings  as  a  causal 
factor  has  been  a  part  of  the  OE  report  form  for  many 
years.  Kinney  ( 1 977b)  indicated  that  inadequate  position 
relief  briefings  were  cited  as  primary  and  contributing 
causes  of  what  were  then  called  system  errors  (now  called 
OEs).  Their  observations  suggested  that  many  position 
relief  briefings  were  too  short.  At  that  time,  he  indicated 
that  specific  rules  for  position  relief  briefings  “. . .  are  not 
in  the  training  curriculum,  not  in  Handbook  71 10.65, 
and  not  on  proficiency  evaluation  checklists,  (p.  6-1).” 
He  also  noted  that  facility  managers  and  supervisors  have 
taken  actions  to  improve  relief  briefings.  Recommenda¬ 
tions  were  provided  to  improve  performance  during 
position  relief  briefings. 

Despite  the  concern  surrounding  position  relief  brief¬ 
ings,  data  extracted  from  the  OE  reports  suggest  that  the 
frequency  with  which  a  position  relief  briefing  is  cited  as 
a  contributing  factor  to  an  OE  is  relatively  small  when 
compared  with  several  other  factors.  Schroeder  (1982), 
in  his  review  of  the  1979  and  1980  version  of  the  OE 
database,  found  that  position  relief  briefings  were  a  direct 
cause  in  0.89%  (1979)  and  0%  (1980)  of  en  route  OEs. 
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For  terminal  operations ,  the  percentages  were  only  slightly 
higher  (1.07%  and  1.06%,  respectively). 

The  later  analysis  of  OEs  conducted  by  the  Office  of 
Aviation  Medicine  in  1988  cited  incomplete  position 
relief  briefings  as  a  common,  error-prone  procedure.  The 
most  recent  annual  report  from  the  NAIMS  (FAA,  2003) 
indicated  that  position  relief  briefings  were  a  factor  in 
12%  of  OEs.  While  an  emphasis  on  position  relief  brief¬ 
ings  has  been  a  frequent  intervention  strategy,  a  survey 
of  controllers,  supervisors,  and  managers  conducted  by 
Mayberry  et  al  (1995)  suggested  that  less  than  10%  of 
controllers  and  supervisors  viewed  position  relief  briefings 
as  one  of  the  top  reasons  for  an  OE.  Analyses  by  Lowry  et 
al.  (2005)  also  found  that  position  reliefbriefings  played  a 
limited  role  in  most  OEs.  While  analyses  of  the  database 
indicate  that  position  reliefbriefings  are  causal  factors  in 
a  small  percentage  of  OEs,  the  consistent  emphasis  on 
initiatives  related  to  this  factor  (which  will  be  discussed 
later)  suggest  that  operational  personnel  believe  this  is 
an  important  factor  in  OEs. 

When  teamwork  addresses  interpersonal  skills,  the  notion 
of  the  team  extends  beyond  a  given  dyad  to  include  sur¬ 
rounding  facility  personnel.  This  extension  is  addressed 
through  documents  about  team  culture  and  plays  a 
prominent  role  in  the  Air  Traffic  Team  Enhancement 
program  (ATTE).  Although  ATTE  was  not  designed 
as  an  OE  mitigation  effort,  per  se,  it  has  been  used  in 
some  cases  to  address  teamwork  as  an  OE  causal  fac¬ 
tor.  Anecdotal  information  from  ten  facilities  suggests 
that  OEs  were  reduced  when  ATTE  was  successfully 
implemented — although  other  variables  confounded 
these  results  and  made  clear  interpretation  difficult.  The 
ATTE  program  is  currently  undergoing  further  refine¬ 
ment  and  testing. 

Efforts  to  determine  the  relationship  between  time-on- 
position  and  the  occurrence  of  an  OE  are  often  consid¬ 
ered  a  part  of  the  position  relief  briefing  issue,  tfowever, 
there  may  be  factors  that  contribute  to  the  occurrence 
of  OEs  shortly  after  assuming  a  position  other  than  the 
nature  and  length  of  the  position  relief  briefing.  In  the 
first  quarter  report  of  the  Air  Traffic  Quality  Assurance 
Staff  (FAA,  1986a),  the  analysis  of  1985  OE  data  revealed 
that  more  than  20%  of  the  OEs  occurred  in  the  first  1 5 
minutes  on  position.  Pounds  and  Ferrante  (2003),  in 
their  review  of  the  1997  to  2000  OE  database,  indicated 
that  9%  of  OEs  occurred  within  the  first  5  minutes, 
18%  in  the  first  10  minutes,  and  35%  within  the  first 
20  minutes  on  position.  Lowry  et  al.  (2005),  in  their 
effort  to  debunk  myths  associated  with  OEs,  found  in 
an  analysis  of  the  NAIMS  OE  database  that  about  15 
to  18%  of  OEs  occurred  in  each  of  the  three  10-minute 


time  periods  during  the  first  30  minutes  on  position. 
The  results  were  similar  for  en  route,  TRACON,  and 
tower  facilities.  They  correctly  pointed  out  that  it  is  dif¬ 
ficult  to  fully  interpret  the  implications  of  these  findings 
without  knowing  the  average  time  on  position.  They  also 
pointed  out  that,  in  their  opinions,  there  were  two  pos¬ 
sible  explanations — that  the  position  reliefbriefings  were 
inadequate  or  that  the  controller  assumed  the  position 
during  a  busy  traffic  period.  A  subsequent  review  of  OE 
narratives  conducted  in  support  of  their  study  tended  to 
refute  the  importance  of  the  position  reliefbriefings,  as 
does  the  previous  literature. 

Our  analysis  of  the  OE  database  suggested  that  traffic 
volume  remains  relatively  consistent  across  time  on  po¬ 
sition.  A  possible  alternative  explanation  to  the  Lowry 
conclusion  is  that  a  portion  of  these  OEs  could  be  at¬ 
tributed  to  the  complexity  associated  with  a  controller 
“getting  the  picture”  or  “getting  up  to  speed”  when 
taking  a  position.  Another  possibility  is  that  we  do  not 
adequately  understand  whether  additional  information 
needs  to  be  communicated  when  a  controller  assumes  a 
position.  In  either  case,  the  data  clearly  demonstrate  that 
the  tendency  for  a  high  percentage  of  OEs  to  occur  early 
on  position  has  been  consistent  for  the  past  two  decades. 
However,  there  is  little  evidence  to  document  the  extent 
to  which  the  position  relief  briefing  is  a  prominent  factor 
in  their  occurrence. 

Training  and  Experience.  Historically,  controllers  learned 
basic  air  traffic  control  procedures  at  the  FAA  Academy 
using  a  generic  airspace.  More  recently,  new  facility  trainees 
have  taken  other  types  of  training  such  as  that  provided 
by  military  facilities  or  College  Training  Initiative  (CTI) 
schools.  When  a  new  trainee  arrived  at  a  facility,  they  were 
called  “developmentals”  whose  skills  are  further  refined 
during  each  facility’s  developmental  training  program. 
Developmental  training  consisted  of  classroom  activities, 
laboratory  simulation  problems,  and  on-the  job  training 
(OJT)  in  which  the  developmental  developed  his/her  skills 
while  under  the  supervision  of  a  formal  OJT  instructor. 
Developmentals  eventually  learned  to  control  traffic 
independently  on  multiple  pieces  of  airspace  (called  “posi¬ 
tions”  in  a  TRACON  and  “sectors”  in  en  route  facilities). 
TRACON  and  en  route  developmental  controllers  also 
learned  to  perform  increasingly  difficult  duties,  starting 
with  entering  flight  data  through  radar  associate  duties 
and  ending  with  radar  duties.  Tower  controller  training 
was  similar  in  that  it  started  with  the  flight  data/ clearance 
delivery  position  but  later  split  into  two  equally  difficult 
positions,  the  ground  and  local  controllers,  that  had  to  be 
learned  separately.  Upon  the  completion  of  field  training, 
a  developmental  became  a  certified  professional  controller 
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(CPC)  by  becoming  certified  on  each  position  required  at 
the  facility  or  area  to  which  he/she  was  assigned.  Because 
of  the  requirement  to  become  certified,  CPCs  can  only 
control  traffic  on  sectors  (in  that  airspace)  on  which  they 
have  been  certified. 

Two  reports,  separated  by  nearly  two  decades,  presented 
similar  findings  about  the  relationship  of  controller 
training  and  experience  to  OEs.  Spahn  (1977)  reported 
that  controllers  with  five  years  or  less  experience  (after 
certification)  were  more  susceptible  to  OEs  than  more 
experienced  controllers.  Mayberry  et  al.  (1995)  qualified 
this  susceptibility  by  stating  that  controllers  with  six  or 
fewer  years  of  experience  were  about  30%  more  likely  to 
have  an  OE  compared  to  more  experienced  controllers. 
Broach  and  Schroeder  (2005),  in  their  analysis  of  en 
route  OE  data  from  1997  to  2003,  found  that  younger 
controllers  with  less  experience  had  the  highest  OE  rates. 
However,  Lowry  et  al.  (2005)  concluded  that  insufficient 
information  was  available  to  draw  a  conclusion  about  the 
relationship  between  experience  and  OEs. 

Findings  such  as  these  suggest  that  greater  attention 
should  be  spent  monitoring  the  performance  of  less 
experienced  controllers.  Though  not  a  research  report, 
Boone’s  (2004)  memo  on  OEs  at  Indianapolis  Center 
(ZID)  suggested  that  a)  a  significant  percentage  of  ZID 
OEs  were  attributed  to  controllers  either  during  OJT  or 
within  two  years  of  certification,  and  b)  when  the  interval 
between  certification  on  all  Radar  Associate  positions  and 
the  commencement  of  Radar  training  was  increased,  that 
percentage  was  reduced. 

Although  there  is  considerable  literature  that  describes  and 
tracks  controller  training  across  time,  there  is  an  absence 
of  literature  on  controller  performance.  For  example,  all 
controllers  certified  to  control  traffic  at  a  given  sector  are 
considered  to  be  equivalent.  Furthermore,  there  appear 
to  be  no  studies  that  have  reported  how  controller  skills 
improve  or  degrade  with  the  passage  of  time. 

Broach  and  Schroeder  (2005)  assessed  error  occurrence 
across  time  for  a  group  of  en  route  controllers.  In  a  fol- 
low-on  analysis,  the  authors  identified  5,559  controllers 
employed  at  an  en  route  center  throughout  the  seven-year 
time  period.  Of  that  group,  3,852  (about  69%)  were 
OE-free,  and  2 1  %  had  a  single  OE.  This  means  that  9% 
of  the  controllers  had  between  two  and  six  OEs  in  seven 
years.  We  do  not  know  how  these  data  extrapolate  across 
the  remaining  years  of  a  controller’s  career,  except  that  the 
overall  OE  rate  tends  to  decline  across  age.  Additional 
information  is  needed  to  understand  what  helped  69% 
of  the  en  route  workforce  to  remain  OE-free  during  this 


time  period.  What  distinguishes  their  performances  from 
those  controllers  who  had  one  or  more  errors? 

There  are  indications  that  controllers  who  made  OEs  may 
perceive  the  error  process  differently  than  those  who  did 
not.  NARI  (Conner  &  Corker,  2001)  found  some  differ¬ 
ences  in  the  perceptions  of  interviewed  controllers  who 
previously  had  zero,  one,  or  multiple  OEs.  For  example, 
controllers  with  no  OEs  considered  ATC  procedure-fol¬ 
lowing  to  be  a  significant  OE  causal  factor,  but  those  with 
multiple  errors  gave  it  the  lowest  weight.  On  the  other 
hand,  controllers  with  no  OEs  rated  ATC  conditions  as 
noncontributory,  while  controllers  with  one  or  multiple 
errors  rated  it  higher. 

HMI  and  Equipment.  Given  the  considerable  expense 
associated  with  equipment  acquisition,  it  is  not  surpris¬ 
ing  that  the  largest  number  of  documents  related  to 
Contextual  Conditions  fell  within  this  category.  For  the 
most  part,  the  documents  that  addressed  OEs  from  an 
HMI  perspective  were  theory-based  because  the  authors 
theorized  about  or  described  the  impact  that  new  and/or 
existing  technologies  have  on  controller  performance  (c.f , 
Hopkin,  1989).  However,  the  effects  of  new  technologies 
on  OEs  were  not  specifically  studied.  Instead,  much  of 
the  focus  was  directed  at  problems  controllers  may  have 
after  the  new  technologies  are  implemented  in  the  field. 
Thus,  the  focus  was  more  on  controller  performance,  in 
general,  rather  than  on  OEs. 

Because  most  of  the  documents  in  this  sub-category  were 
associated  with  new  technologies,  it  was  disappointing  not 
to  find  documents  on  field  studies  that  tracked  the  impact 
that  the  technologies  had  on  OEs.  While  it  is  known  that 
during  the  early  phase  of  field  implementation,  some 
controllers  incur  an  OE  due  to  HMI  problems  (typically, 
when  this  happens  the  equipment  and  not  the  controller 
is  charged  with  the  OE) ;  once  the  problems  are  addressed, 
it  appears  that  further  evaluation  is  not  conducted  of 
the  impact  of  the  technologies  on  OEs.  One  exception 
was  a  concern  raised  by  Corker  (2004)  about  potentially 
negative  effects  resulting  from  changes  in  procedures  and 
utilization  of  the  new  URET  display. 

If  a  new  technology  is  expected  to  have  a  marked  impact 
on  OE  reduction,  then  once  implemented,  there  should 
be  a  discontinuity  in  the  OE  error  rates  between  pre- 
and  post-  implementation.  Despite  the  relatively  simple 
analyses  needed  to  conduct  a  pre-  and  post-  implemen¬ 
tation  study,  few  studies  have  analyzed  the  impact  that 
new  technologies  have  had  on  the  incidence  of  OEs.  As 
the  ATO  continues  with  its  modernization  program,  it  is 
critical  that  that  the  appropriate  baseline  data  are  being 


18 


collected  to  ensure  a  proper  assessment  of  a  technology’s 
positive  or  negative  effect  on  controller  performance. 

Non-Compliance  Issues 

Non-compliance  is  a  purposeful  disregard  of  rules  or 
procedures,  existing  rules,  regulations,  instructions,  or 
standard  operating  procedures  by  choosing  not  to  respond 
or  selecting  an  action  not  in  compliance  with  rules  or 
operating  procedures.  Factors  related  to  non-compliance 
with  ATC  policies  and  procedures  were  not  the  focus  of 
any  of  the  OE  literature  that  we  reviewed,  and  conse¬ 
quently  will  not  be  discussed  in  this  paper. 

Task  Descriptions 

A  task  is  an  activity  that  a  controller  is  primarily  engaged 
in  at  a  specific  time.  Controllers  often  multitask,  i.e., 
perform  several  tasks  at  a  time.  The  controller  task  being 
performed  at  the  time  of  an  OE  was  not  the  focus  of  any  of 
the  OE  literature  we  reviewed,  and  consequently  will  not 
be  discussed  in  this  paper.  However,  the  OE  report  forms 
categorize  aspects  of  controller  activities,  for  example, 
monitoring  the  radar  display  or  observing  traffic. 

Response  Execution 

Response  execution  refers  to  the  outputs  or  actions  based 
on  the  earlier  processes,  that  is,  how  choices  and  plans 
were  enacted  as  actions  (either  verbally  or  manually). 
Response  execution  errors  are  often  characterized  by 
their  timing  (such  as  too  soon,  too  late)  or  information 
quality  (such  as  unclear  or  incorrect).  They  can  also  be 
classified  as  errors  of  omission  (absent)  or  commission 
(performed). 

Response  execution  appears  closely  associated  with  con¬ 
troller  planning  and  decision  making.  That  is,  response 
execution  is  the  result  of  either  planned  or  unplanned 
actions  resulting  from  controller  decisions  based  on 
factors  that  affect  those  decisions,  including  percep¬ 
tion  and  vigilance,  memory,  contextual  conditions,  and 
organizational,  managerial,  and  supervisory  influences. 
Consequently,  instead  of  discussing  reports  about  con¬ 
troller  actions  that  resulted  in  OEs,  in  this  section,  we 
discussed  those  actions  as  they  were  relevant  to  the  other 
JANUS  categories. 

Mental  Processes 

Thirty-two  percent  of  the  OE-related  documents  fell  in 
the  JANUS  Mental  Processes  category.  Figure  7  shows 
the  distribution  of  documents  classified  into  the  JANUS 


sub-categories  for  Mental  Processes.  Two  sub-categories 
accounted  for  over  three-fourths  of  the  reports:  Percep¬ 
tion  and  Vigilance  (47%),  and  Memory  (32%).  The 
literature  from  these  sub-categories  will  be  discussed  in 
some  detail.  For  a  list  of  the  reports  classified  as  belonging 
to  the  remaining  JANUS  mental  processes  subcategories, 
the  reader  is  referred  to  Appendix  B. 

Figure  7  shows  the  distribution  of  the  OE  literature  based 
on  the  mental  processes  addressed.  Most  of  the  mental 
processing  OE  literature  focused  on  aspects  of  perception 
and  vigilance,  followed  by  controller  memory,  planning 
and  decision-making,  then  response  execution.  Although 
executing  a  response  is  not  typically  thought  of  as  a  mental 
process,  responses  are  included  here  as  consequences  of 
mental  processes. 

Perception  and  Vigilance.  Perception  involves  acquiring 
and  processing  sensory  information  to  hear,  see,  touch, 
taste,  and  smell  objects  in  the  environment.  Vigilance  is 
the  maintenance  of  attention  required  to  obtain  infor¬ 
mation  that  may  be  presented  infrequently  or  monitor 
a  dynamic  and  ever-changing  environment. 

In  the  OE  literature,  perception  and  vigilance  have  often 
been  studied  under  the  name  of  controller  attention, 
(Kinney,  1977b;  Schroeder,  1983),  controller  awareness 
and  attention  (FAA,  1986a;  Rodgers  &  Nye,  1993),  and 
later  SA  (Endsley  &  Rodgers,  1994;  Rodgers,  Mogford,  & 
Strauch,  2000;  Kelly,  Krantz  &  Spelman,  2001).  Studies 
designed  to  determine  the  effectiveness  of  OE  interven¬ 
tions  tend  to  rely  on  surveys,  interviews,  and  self-reports 
from  controllers  and  are  lacking  in  the  use  of  variables 
that  measure  controller  perception  and  vigilance. 

Estimates  of  the  percentage  of  OEs  related  to  percep¬ 
tion  and  vigilance  vary  from  a  low  of  27%  (Kelly  et 
al.,  2001)  to  a  high  of  72%  (Rodgers,  Mogford,  & 
Strauch,  2000).  The  variation  in  statistics  is  due  to  the 
type  of  data  studied  (OE  narratives  vs.  causal  factors 
identified  on  the  OE  final  report)  and  the  time  period 
over  which  the  evaluation  was  conducted.  Although 
controller  perception  and  vigilance  have  been  often 
studied,  a  coordinated  effort  over  time  to  track  OEs 
specifically  related  to  a  loss  or  lack  of  perception  and 
vigilance  has  been  impeded  because  the  OE  reporting 
form  did  not  include  these  causal  factors.  To  address 
this  gap,  Rodgers  &  Nye  (1993)  recommended  the 
addition  of  three  causal  factors  to  the  OE  final  report 
to  track  the  amount  of  controller  SA:  a)  failure  to 
detect  displayed  information,  b)  failure  to  compre¬ 
hend  displayed  information,  and  c)  failure  to  project 
future  status  of  displayed  data.  Similarly,  in  the  tower 


19 


Figure  7.  Distribution  of  OE-related  reports  by  JANUS  Mental  Processes  sub-categories. 


environment,  failure  to  observe  aircraft  might  also 
be  included  as  an  indicator  of  lack  of  perception  and 
vigilance  (Kelly  et  al.,  2001). 

Memory.  Memory  consists  of  Working  Memory  (WM) 
and  Long-term  Memory  (LTM).  WM  is  a  “blackboard” 
or  “scratchpad”  where  one  creates  and  maintains  an  un¬ 
derstanding  of  a  situation.  WM  is  defined  as  a  processing 
resource  of  limited  capacity  involved  in  the  preservation 
of  information  while  simultaneously  processing  the 
same  or  other  information.  LTM  is  the  “library”  of  one’s 
learned  information  and  accumulated  experiences.  The 
LTM  system  is  responsible  for  storing  information  on  a 
relatively  permanent  basis. 

Although  memory  failures  have  been  associated  with  OEs 
in  the  en  route,  TRACON,  and  tower  environments  (for 
a  review  of  the  literature  see  Vingelis  et  al.,  1990),  much 
of  the  memory  literature  relevant  to  OEs  has  been  in 
the  context  of  runway  incursions.  In  1986,  the  NTSB 
published  a  special  report  on  runway  incursions  that  con¬ 
cluded  that  the  primary  controller-related  factors  associ¬ 
ated  with  runway  incursions  were  forgetting  aircraft  and 
absent  or  incomplete  coordination  between  controllers. 
In  2001,  Cardosi  and  Yost  reviewed  the  literature  and 
analyzed  safety  data  involving  controller  errors  in  airport 
tower  operations.  Their  review  and  analyses  confirmed 
previous  findings  that  identified  memory  failures  as  the 
most  frequently  cited  causal  factor  of  tower  OEs.  The 
second  most  common  factor  was  communication  errors, 
followed  by  coordination  errors. 


Memory  aids  are  installed  in  all  FAA  facilities,  although 
there  appears  to  be  no  standardization  in  their  use. 
Each  facility  is  left  to  their  own  resources  to  design  and 
implement  effective  memory  aids.  While  the  lack  of 
standardization  leaves  room  for  error,  the  added  flex¬ 
ibility  enables  facilities  to  create  memory  aids  unique  to 
their  environment  (e.g.  addressing  a  particular  runway 
configuration  problem). 

In  2003,  the  Department  of  Transportation  OIG  ac¬ 
knowledged  that  the  FAA’s  numerous  initiatives  directed 
at  reducing  runway  incursions,  including  the  effective 
use  of  memory  aids,  have  been  successful.  As  the  amount 
of  traffic  in  the  NAS  is  returning  to  pre-  September  1 1, 
2001  levels,  the  number  of  runway  incursions  has  steadily 
been  declining.  However,  it  is  unclear  whether  any  of 
the  many  FAA  initiatives  have  been  responsible  for  the 
decline.  Nevertheless,  given  the  predominant  role  that 
memory  failures  have  played  in  tower  OEs,  it  is  likely 
that  memory-related  OEs  have  also  declined. 

Planning  and  Decision  Making.  Planning  and  Decision 
Making  consists  of  four  types  of  mental  processes:  (1) 
Planning  refers  to  the  development  of  a  stepwise  process 
to  reach  an  outcome.  (2)  Decision  making  occurs  when 
more  than  one  option  is  considered  and  then  a  choice 
between  them  is  made.  (3)  When  one  solution  emerges 
from  one’s  understanding  of  the  situation,  this  is  called 
a  judgment.  (4)  Problem  solving  occurs  when  a  solution 
is  needed  to  resolve  a  situation  or  reach  a  goal,  but  no 
ready  options  are  available. 
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Earlier,  when  discussing  the  type  of  studies,  we  noted 
that  the  experiments  reviewed  did  not  use  OEs  as  the 
criteria  of  interest.  Instead,  the  intent  of  the  studies 
was  to  investigate  factors  that  affected  OE  precursors, 
such  SA,  memory,  workload,  and  communications.  As 
discussed,  the  problem  with  conducting  experiments,  in 
general,  is  that  OE  precursors  do  not  operate  in  isola¬ 
tion.  Instead,  they  interact  in  complex  ways  to  increase 
the  vulnerability  of  a  controller  to  committing  an  OE. 
That  very  same  complexity  is  also  true  with  planning 
and  decision-making,  which  is  the  result  of  the  complex 
interaction  of  memory,  perception,  and  vigilance  (D’Arcy 
&  Della  Rocco,  2001). 

The  focus  of  the  FAAs  modernization  efforts  has  been 
directed  at  providing  controllers  with  improved  decision- 
support  tools;  however,  there  appear  to  be  no  studies 
that  examine  the  effectiveness  of  operational  decisions. 
Herein  lies  the  problem.  Although  various  cognitive 
models  have  been  proposed  to  elucidate  controller  deci¬ 
sion-making  (Wickens,  Mavor,  &  McGee,  1997),  these 
models  have  not  been  used  to  validate  the  effectiveness 
of  any  decision  support  tool.  For  example,  in  the  model 
proposed  by  Wickens  et  al.,  controller  memory  serves  as 
the  foundation  for  SA,  which  in  turn  affects  the  quality 
of  the  controller’s  decision.  If  the  underlying  foundation 
is  faulty  (due  to  memory  failure  or  incomplete  SA),  the 
decision  must  be  faulty. 

When  assessing  the  effectiveness  of  a  particular  decision 
support  tool,  it  is  necessary  to  determine  how  the  tool 
is  integrated  into  the  overall  model  of  the  controller’s 
mental  process.  Moreover,  once  a  decision  support  tool  is 
developed,  it  is  important  to  track  its  effect  on  reducing 
OEs  that  were  the  result  of  faulty  decision-making. 

FAA  Initiatives  to  Reduce  Operational  Errors 

The  FAA  has  a  history  of  implementing  OE  reduction 
initiatives  developed  at  various  levels  within  the  legacy 
AAT  and  current  ATO  organizations.  Ongoing  analyses 
of  OEs  by  the  FAA  revealed  recurring  causal  factors,  and 
initiatives  were  periodically  fielded  to  address  them.  Top¬ 
ics  have  focused  on  communicating  about  OE  trends  to 
increase  awareness  of  OE  causal  factors  by  all  personnel 
in  the  organization,  developing  clear  performance  stan¬ 
dards,  ensuring  compliance  to  standards,  and  developing 
long-term  strategies. 


Examination  of  initiatives  helps  to  show  where  the  ATO 
has  focused  its  energies.  This  section  is  organized  in  a 
similar  manner  to  the  previous  section  that  illustrated 
where  the  research  community  focused  its  energies.  If 
there  is  a  disconnect  between  the  two  groups,  we  can 
speculate  about  why  it  might  have  resulted;  for  example, 
because  the  organization  was  not  aware  of  research  activi¬ 
ties,  the  researchers  were  not  aware  of  operational  needs, 
or  perhaps  the  research  results  were  not  considered  use¬ 
ful  for  operational  application.  Whatever  the  reasons, 
if  there  are  gaps  between  the  efforts  of  the  research  and 
operational  communities,  then  steps  should  be  taken  to 
bridge  them. 

To  examine  OE-reduction  initiatives,  we  used  any  avail¬ 
able  information  that  described  a  program  intended  to 
reduce  OEs.  Materials  included  such  items  as  FAA  orders 
and  memoranda,  briefing  materials,  and  meeting  notes. 
Because  we  were  unable  to  locate  a  historical  archive  of 
these  initiatives,  most  included  here  are  fairly  recent. 
Nevertheless,  we  feel  confident  that  similar  initiatives 
were  previously  undertaken,  due  to  the  consistent  role 
of  various  factors  in  OE  causation. 

We  identified  222  initiatives  to  classify.  Because  many 
of  the  documents  were  not  dated,  it  was  not  always 
possible  to  place  a  given  initiative  within  a  historical 
context.  Furthermore,  it  was  not  possible  to  trace  the 
interconnections  among  source  materials  (i.e.,  whether 
one  document  was  related  to  another);  thus,  some  of  the 
222  initiatives  probably  represent  duplicate  activities. 
When  a  given  document  contained  multiple  initiatives, 
each  initiative  was  classified  into  its  respective  JANUS 
category.  Thus,  the  number  of  initiatives  does  not  reflect 
the  number  of  documents  examined.  Furthermore,  when 
a  given  initiative  addressed  several  JANUS  categories, 
then  the  initiative  was  included  in  multiple  categories. 
For  example,  one  facility’s  “effective  hearback”  com¬ 
petition  to  highlight  the  frequency  and  significance  of 
hearback/ readback  issues  was  classified  in  the  following 
JANUS  sub-categories:  Perception  and  Vigilance,  Pilot- 
Controller  Communications,  and  Training  and  Experi¬ 
ence.  Finally,  the  classification  of  OE  reduction  initiatives 
was  based  on  the  topics  addressed  and  not  the  quality 
of  the  implementation.  Concerning  the  latter,  in  most 
cases,  due  to  a  lack  of  documentation,  it  was  not  pos¬ 
sible  to  differentiate  between  an  initiative  that  was  only 
planned  and  one  that  was  actually  implemented.  Thus, 
initiatives  that  were  either  planned  and  implemented 
or  planned  but  not  implemented  are  included  in  the 
JANUS  classification. 
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With  the  above  caveats  in  mind,  Figure  8  shows  the  fre¬ 
quencies  of  initiatives  targeting  each  JANUS  category. 

Most  initiatives  in  this  sample  targeted  contextual  condi¬ 
tions  that  contribute  to  human  performance  and  result 
in  OEs.  This  category  was  further  broken  down  into  its 
subcategories  (see  Figure  9).  Initiatives  classified  in  the 
Organization  and  Management  categories  reflect  activities 
intended  to  be  accomplished  by  headquarters  or  regional 
personnel,  such  as  establishing  a  method  for  distributing 
information  about  OEs  to  facilities. 

Figure  9  shows  that  most  of  the  initiatives  introduced  by 
the  Air  Traffic  Organization  involved  controller  training, 
teamwork,  and  communications.  Other  initiatives  not 
captured  in  these  categories  were  related  to  Performance 
Management/  Performance  Standards  (n=12)  and  Con¬ 
troller-Ground  Communications  (n=l). 

Figure  10  shows  the  number  of  initiatives  targeting  the 
JANUS  Mental  Processes  subcategories.  The  highest 
percentage  of  initiatives  targeted  Perception  and  Vigilance 
(59%),  and  about  half  that  many  targeted  Planning  and 
Decision  Making  (29%).  About  17%  of  the  Mental 
Processes  initiatives  targeted  Memory,  and  only  about 
7%  targeted  Response  Execution. 

Figure  1 1  compares  the  Contextual  Conditions  catego¬ 
rizations  of  both  the  research  documents  and  initiatives. 
Given  the  possibility  that  some  OE  reduction  initiatives 
are  duplications,  comparisons  made  between  OE  research 
and  OE  initiatives  should  focus  on  the  trends  and  not 


the  number  of  initiatives  in  a  given  JANUS  category.  It 
is  apparent  that  the  same  six  contextual  conditions  were 
emphasized  by  both  the  research  and  OE  reduction 
initiatives  (although  not  necessarily  in  the  same  order): 
a)  training  and  experience,  b)  teamwork,  c)  pilot-ATC 
communications,  d)  FIMI  and  equipment,  e)  airspace/ 
surface,  and  f)  traffic.  Although  the  specific  topics  ad¬ 
dressed  by  research  and  the  OE  reduction  interventions 
may  differ,  the  trend  comparisons  suggest  that  research 
and  operations  focused  on  the  same  general  areas. 

Figure  12  compares  the  distributions  of  initiatives  and 
research  articles  targeting  the  OE  Mental  Processes  sub¬ 
categories.  Again,  when  the  trends  are  compared,  we  see 
that,  with  the  exception  of  memory,  both  research  and 
OE  reduction  initiatives  had  a  similar  focus  on  the  mental 
processes.  While  research  focused  on  memory  more  often 
than  did  the  OE  reduction  initiatives,  this  should  not 
be  misconstrued  as  a  lack  of  operational  concern  with 
regard  to  memory.  On  the  contrary,  addressing  control¬ 
ler  memory  issues  continues  to  be  of  major  importance 
in  ATC  operations  and  is  evident  in  the  wide  range  of 
memory  aids  employed  at  ATC  facilities. 

A  brief  overview  of  the  initiatives  is  given  here.  Appendix 
C  contains  more  detailed  descriptions  of  several  of  the 
initiatives,  including  air  traffic  re-creation  tools,  video 
briefing  materials,  the  National  Air  Traffic  Professional¬ 
ism  program,  facility  initiatives,  activities  detailed  in 
the  Air  Traffic  Quality  Assurance  Order,  7210.56  (FAA, 
2002b),  initiatives  in  the  FAA/NATCA  3-Year  Plan  for 
OE  Reduction  (FAA,  2002c),  and  activities  in  the  Short 
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Figure  8.  Distribution  of  OE-related  initiatives  by  JANUS  category. 
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Figure  9.  Initiatives  targeting  JANUS  OE  Contextual  Conditions  subcategories. 
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Figure  10.  Initiatives  targeting  JANUS  OE  Mental  Processes  subcategories. 
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□  Initiatives  □  Research 


Figure  1 1 .  Comparison  of  results  from  the  categorization  of  research  articles  and  initiatives  related  to 
Contextual  Conditions. 
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Figure  12.  Initiatives  and  research  articles  targeting  subcategories  of  mental  processes. 
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Term  OE  Reduction  Initiative  (FAA,  2002a).  Examples 
of  recommendations  from  workgroups  are  also  included, 
such  as  those  from  the  Operational  Error  Prevention 
and  Reduction  Workgroup,  which  met  in  1996,  and  the 
Operational  Errors  in  the  U.S.  Air  Traffic  System  —  A 
Task  Group  Review  (FAA,  1987). 

Initiatives  Classified  by  JANUS  Categories 

Organization  and  Management 

Recall  that  organizational  and  management  initiatives 
should  be  incorporated  into  national/  facility  level  poli¬ 
cies  and  procedures.  Elowever,  the  resources  dedicated  to 
the  implementation  of  national/facility  level  policies  and 
procedures  are  equally  important.  For  example,  the  QA 
programs  detailed  in  FAA  Order  7210.56  (FAA,  2002b) 
correspond  to  initiatives  that  fall  within  the  JANUS 
Organization  and  Management  tiers. 

At  the  national  level.  The  Air  Traffic  Evaluations  and 
Investigation  Manager  and  Staff  are  expected  to  provide 
the  necessary  guidance  to  ensure  that  the  QA  program 
succeeds.  Among  other  things,  this  guidance  includes 
evaluating  facilities’  ATC  QA  programs  and  recognizing 
QA  programs  that  have  achieved  significant  accomplish¬ 
ments  in  OE  reduction.  For  example,  when  a  facility  has 
achieved  1,000,000  error-free  operations,  it  is  awarded 
a  national  certificate  of  recognition  as  a  member  of  the 
“None  in  the  Million”  Club. 

At  the  regional  level,  AirTraffic  Division  (ATD)  Managers 
are  expected  to  develop  a  regional  QA  program,  which 
includes  a  regional  OE/OD  prevention  plan.  The  plan 
is  included  in  a  Regional  QA  Order  that  is  then  used  to 
develop  corresponding  Facility  QA  programs  and  asso¬ 
ciated  facility  OE/OD  prevention  plans.  Paralleling  the 
national  evaluation  process,  ATD  Managers  are  required 
to  conduct  annual  reviews  of  existing  regional  QA  orders 
and  programs.  They  also  ensure  that  approved  facility 
OE/ OD  prevention  plans  are  in  effect  that  emphasize  the 
use  of  the  basic  ATC  procedures  to  prevent  and  reduce 
OE/ODs  and  address  the  items  pertinent  to  a  particular 
facility’s  past  deficiencies. 

However,  none  of  the  documents  we  reviewed  indicated 
whether  adequate  resources  were  provided  to  accom¬ 
plish  the  initiatives.  And,  in  spite  of  the  existence  of  a 
mechanism  that  should  provide  a  detailed  assessment  of 
OE/OD  reduction  initiatives,  we  were  unable  to  find 
documentation  about  the  success  or  failure  of  most  re¬ 
gional  or  facility-level  OE/ OD  reduction  initiatives.  This 


does  not  mean  that  the  information  was  not  collected. 
To  the  contrary,  it  is  likely  that  the  information  resides 
somewhere  within  the  ATO  at  the  national,  regional, 
and/or  facility  level.  For  example.  Appendix  C  includes 
descriptions  of  Houston  ARTCC’s  Quality  Evaluation 
Program  (QEP),  Kansas  City  ARTCC’s  Performance 
Management  Program,  and  the  analysis  of  the  Great  Lakes 
Region’s  OE  Reduction  Efforts.  Each  of  these  initiatives 
included  an  evaluation.  Overall,  however,  there  appears 
to  be  no  central  repository  for  this  kind  of  information, 
which  makes  it  difficult  to  conduct  a  national  audit  to 
determine  the  success  of  the  various  OE  reduction  initia¬ 
tives  implemented  at  different  sites. 

Supervision 

Operational  supervisors  (first-line  supervisors)  are  impor¬ 
tant  to  the  success  of  a  facility’s  OE  reduction  program. 
Supervisors  are  tasked  to  conduct  both  primary  opera¬ 
tional  duties  and  administrative  duties.  They  monitor  the 
operations  and  manage  controller  performance.  However, 
few  initiatives  indicated  whether  there  was  organizational 
support  for  supervisors  to  accomplish  these  tasks.  One 
task  group  that  reviewed  the  air  traffic  system  (FAA, 
1987)  raised  a  similar  question.  Noting  that  the  quality 
of  supervision  varied  both  within  and  between  facilities, 
the  task  group  recommended  that  greater  attention  should 
be  placed  on  supervisory  training.  Although  first-line 
supervisors  come  from  the  ranks  of  controllers,  this  does 
not  necessarily  mean  that  their  technical  skills  prepare 
them  to  manage  controller  performance  and  training. 
Good  controller  skills  do  not  automatically  translate  into 
good  supervisory  skills. 

Contextual  Factors 

Most  OE  reduction  initiatives  that  addressed  contextual 
factors  focused  on  three  areas:  (1)  training  and  experience, 
(2)  teamwork,  and  (3)  pilot-ATC  communications. 

Training  and  Experience.  Skill  development  acquired 
through  training  and  experience  has  been  an  ongoing 
focus  of  OE  reduction  initiatives.  Initiatives  include 
such  things  as  designing  suitable  training  platforms 
(e.g.  computer-based  instruction  and  ATC  simulators), 
developing  course  content  to  address  skill  deficiencies 
associated  with  specific  kinds  of  OEs  (i.e.,  those  associ¬ 
ated  with  errors  in  data  posting,  radar  display,  aircraft 
observation,  communication,  coordination,  and  posi¬ 
tion  relief  briefings),  and  identifying  the  types  of  work 
experiences  necessary  to  certify  controllers  and  ensure 
that  they  maintain  currency. 
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Although  numerous  types  of  ATC  simulators  have  been 
developed,  most  have  been  used  for  the  training  of  novice 
controllers.  However,  the  need  to  use  ATC  simulators 
for  proficiency  training  of  experienced  controllers  was 
raised  during  the  1987  task  group  that  reviewed  the  air 
traffic  system  (FAA,  1987),  again  in  the  Short  Term  OE 
Reduction  Initiative  (FAA,  2002a),  and  more  recently  in 
the  FAA’s  2005-2009  Flight  plan.  To  date,  few  initiatives 
have  been  developed  to  address  these  issues  for  experi¬ 
enced  controllers. 

Teamwork.  OE-reduction  initiatives  related  to  teamwork 
primarily  focused  on  two  topics:  1)  the  exchange  of 
information  among  team  members  and  2)  developing  a 
positive  approach  to  teamwork.  Over  a  1 4-year  span,  from 
the  1987  task  group  that  reviewed  the  air  traffic  system 
(1987)  to  the  2002  Short  Term  OE  Reduction  Initiative 
(FAA,  2002a),  the  focus  of  information  exchange  initia¬ 
tives  was  on  the  position  relief  briefing  and  the  use  of 
flight  strips.  Initiatives  designed  to  improve  the  concept 
of  teamwork  among  controllers  followed  the  air  traffic 
team  enhancement  model  and  its  emphasis  on  profes¬ 
sionalism  and  respect  for  each  other  (FAA,  1987;  FAA, 
2002a).  It  should  be  noted  that,  although  components  of 
teamwork  (i.e. ,  position  relief  briefing  and  coordination) 
have  been  identified  as  OE  causal  factors,  deficiencies  in 
teamwork  have  not.  Thus,  if  the  emphasis  on  teamwork 
as  an  OE  mitigation  strategy  continues,  greater  attention 
needs  to  be  placed  on  identifying  the  specific  aspects  of 
teamwork  that  affect  OE  occurrence,  developing  specific 
interventions,  and  tracking  their  effectiveness. 

Pilot-ATC  Communications.  Initiatives  designed  to  re¬ 
duce  hearback/readback  errors  are  ever-present  across 
time  (Cardosi,  2001;  FAA,  1987,  2002a  and  2002b). 
Past  initiatives  have  included  developing  training  vid¬ 
eos  and  materials  based  on  OE  case  histories,  auditing 
pilot-controller  communications,  and  designing  pilot- 
controller  training.  Given  all  the  attention  directed 
at  reducing  OEs  related  to  hearback/readback  errors, 
one  wonders  whether  any  of  these  initiatives  reduced 
OEs  in  either  the  short-term  (when  awareness  was 
high)  or  the  long-term  (when  the  topic  faded  into 
the  background  and  another  awareness  program  was 
fielded).  Without  clear  evaluations  of  initiatives  related 
to  hearback/readback  errors,  it  is  unclear  what  can  be 
done  to  further  reduce  this  type  ofhuman  error — except 
perhaps  through  technological  advancement  in  digital 
communications.  However,  the  inability  to  further 
reduce  hearback/readback  errors  is  merely  an  assump¬ 
tion  until  more  information  is  known  about  these  types 
of  performance  errors,  including  information  about 


how  they  occur  during  normal  operations  and  what 
circumstances  contribute  to  some  of  these  performance 
errors  becoming  OEs. 

Task  Descriptions 

Some  initiatives  addressed  the  type  of  task  the  control¬ 
ler  was  performing  if  it  was  noted  as  being  performed  at 
the  time  of  the  OE.  For  example,  improving  processes 
associated  with  the  transfer  of  position  responsibility  re¬ 
ceived  repeated  attention  because  of  the  increase  in  OEs 
reported  within  20  minutes  of  a  position  relief  briefing. 
Topics  related  to  the  task  of  communicating  with  pilots 
also  received  attention  under  topics  such  as  improving 
phraseology  and  reducing  readback/hearback  errors. 

Non-Compliance  Issues 

National,  regional,  and  local  SOPs  were  established  and 
audits  were  conducted  to  ensure  that  facility  directives  were 
consistent  with  national  Orders.  Controller  non-compli¬ 
ance  with  SOPs  was  not  addressed  by  major  initiatives, 
other  than  those  that  emphasized  “back  to  basics.” 

Response  Execution 

No  major  OE  reduction  initiatives  specifically  targeted 
factors  related  to  how  responses  were  executed  (e.g.,  the 
action  taken  was  not  timely  or  the  wrong  entry  was  made 
on  the  keyboard).  Response  execution  errors  were  often 
left  for  specific  training  programs  related  to  new  systems 
(e.g.,  STARS,  URET)  to  address. 

Mental  Processes 

As  was  stated  earlier  in  this  report,  the  controller’s 
job  is  primarily  a  mental  one  and,  thus,  one  would 
expect  a  large  number  of  initiatives  to  be  developed 
to  address  controller  mental  processes  such  as  percep¬ 
tion  and  vigilance,  memory,  and  planning  and  deci¬ 
sion-making.  However,  this  has  not  been  the  case. 
Instead,  most  of  the  mental  processing  initiatives  we 
reviewed  were  associated  with  the  training  of  novice 
controllers.  When  mental  processes  were  addressed 
for  experienced  controllers,  the  initiatives  tended  to 
be  more  of  the  awareness-related  refresher  training. 
What  appear  to  be  lacking  are  initiatives  that  allow 
experienced  controllers  to  test  their  mental  processing 
skills  and  participate  in  activities  designed  to  improve 
those  skills.  One  initiative  that  best  illustrates  the  latter 
approach  is  the  National  Air  Traffic  Professionalism 
(NATPRO)  program. 
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NATPRO  is  a  personal  performance  enhancement  pro¬ 
gram  intended  to  maintain  skilled  performance  rather 
than  provide  remedial  training  for  OE  reduction.  Instead 
of  relying  solely  on  knowledge-based  awareness  training, 
this  approach  integrates  the  concept  of  “performance 
coaching”  by  facility  personnel  certified  as  coaches.  The 
program  combines  knowledge  with  practice  so  control¬ 
lers  can  experience  their  own  strengths  and  limitations 
and  can  gain  insight  into  their  mental-processing  skills, 
especially  those  used  in  air  traffic  control.  These  skills 
include  perception  and  vigilance,  memory,  and  plan¬ 
ning  and  decision-making,  all  of  which  are  practiced  in 
a  dynamic  environment. 

NATPRO  is  designed  as  a  series  of  installments  that 
continue  to  challenge  participants  toward  enhancement 
of  skills  associated  with  mental  processing.  Series  1  of  the 
current  NATPRO  installment  emphasizes  perception  and 
vigilance  through  the  visual  acquisition  of  information. 
The  skills  developed  include  multi-tasking,  paying  atten¬ 
tion,  scanning,  detecting  information,  focusing  on  relevant 
information,  and  staying  “in  your  zone.”  Although  the 
skills  are  generic,  it  is  expected  that  by  improving  these 
generic  skills,  individuals  will  demonstrate  a  corresponding 
improvement  in  job  performance.  By  testing  themselves 
against  the  computer  and  experiencing  how  their  per¬ 
formance  can  vary  in  relation  to  such  factors  as  distrac¬ 
tion,  fatigue,  and  boredom,  participants  gain  increased 
understanding  of  their  own  performance  and  can  develop 
personal  improvement  strategies.  It  will  be  necessary  to 
investigate  whether  such  personal  improvement  strategies 
will  lead  to  improvements  in  controller  performance  under 
simulated  and  actual  job  conditions. 

To  the  extent  that  we  can  measure  differences  in  controller 
performance,  we  have  a  means  of  measuring  the  effec¬ 
tiveness  of  these  types  of  skill  enhancement  programs. 
Developing  controller  performance  measures  would  be  a 
much  better  alternative  than  using  the  number  and  severity 
of  OEs  as  a  performance  metric  for  individuals. 

Discussion  of  Initiatives 

Available  information  about  past  and  current  initiatives 
shows  that  the  FAA  has  a  history  of  deploying  programs 
to  reduce  OEs  that  appear  to  revisit  many  of  the  same 
general  causal  factors.  One  reason  that  the  FAA  continues 
to  implement  the  same  strategies  over  and  over  may  be 
that  the  factor  of  interest  was  not  affected  by  previous 
initiatives  (or  was  only  temporarily  affected  until  the 
initiative  was  concluded)  and  continues  to  be  identified 
as  an  OE  causal  factor.  Furthermore,  air  traffic  control 
is  a  complex  domain,  and  perhaps  simplistic  strategies 


for  reducing  OEs  are  not  effective.  A  third  reason  why 
the  FAA  continues  to  repeat  the  same  strategies  might 
be  that  those  who  develop  new  initiatives  “go  with  what 
they  know”  and  hope  it  will  work  “this  time.”  A  fourth 
reason  might  be  that  personnel  who  develop  new  OE 
initiatives  are  new  to  the  domain  and  have  no  institutional 
recollection  of  programs  that  have  been  tried  previously. 
Naivete  and  a  lack  of  a  historical  record  can  lead  one  to 
select  the  “low  hanging  fruit.”  A  fifth  reason  might  be 
that  personnel  who  develop  OE  initiatives  think  their 
program  will  be  more  effective  at  addressing  the  factor 
than  previous  programs.  It  is  not  productive  to  repeat  the 
same  action  while  expecting  a  different  outcome. 
Whatever  the  reason,  the  content  and  outcomes  of  OE 
reduction  initiatives  tend  to  be  forgotten  because  they 
are  not  sufficiently  documented.  Repeating  unsuccessful 
OE  reduction  initiatives  should  be  avoided  as  a  waste 
of  resources.  One  way  to  decrease  the  likelihood  of  re¬ 
peating  previously  unsuccessful  initiatives  would  be  to 
maintain  an  archive  of  all  OE  reduction  initiatives  (at 
the  national,  regional,  and  facility  levels)  and  document 
their  outcomes  so  that  future  developers  would  not  imple¬ 
ment  unsuccessful  programs.  However,  it  is  also  unclear 
how  to  identify  successful  initiatives.  Opinions  about  an 
initiative’s  effectiveness,  for  example,  although  relatively 
easy  to  obtain  (e.g.,  surveys,  anecdotes,  focus  groups)  have 
all  the  disadvantages  of  subj  ective  data.  Ontheother  hand, 
using  OE  counts  or  rates,  which  are  infrequent  outcomes 
describing  very  complex  situations,  may  not  be  sensitive 
enough  to  evaluate  whether  the  improvement  in  any  one 
factor  resulted  in  a  decrease  in  OEs. 

SUMMARY  AND  CONCLUSIONS 

Our  literature  review  was  part  of  a  larger  effort  to  iden¬ 
tify  more  immediate  short-term  strategies  that  could  be 
undertaken  to  reduce  the  recent  increase  in  OEs.  We  also 
anticipated  that  the  effort  would  identify  potential  new 
intervention  strategies  as  well  as  gaps  in  our  OE  knowledge 
base  that  require  additional  research.  We  gathered  and 
reviewed  documents  that  described  previous  OE  reduction 
initiatives  in  an  effort  to  identify  strategies  that  may  have 
been  successful  in  the  past  and  identify  dimensions  that 
could  benefit  from  interventions,  based  on  the  research 
outcomes.  To  structure  our  approach  to  identifying  gaps  in 
the  literature,  we  used  the  JANUS  taxonomy  to  understand 
the  extent  to  which  HF  issues  typically  associated  with  hu¬ 
man  errors  had  been  addressed  by  research  on  OEs.  This 
assessment  revealed  that  a  majority  of  the  literature  was 
focused  on  contextual  conditions  and  mental  processes. 
We  then  described  the  findings  about  contextual  condi¬ 
tions  and  mental  processes  that  figured  most  prominently 
in  the  literature. 
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As  we  begin  our  discussion  of  the  gaps  in  the  literature, 
it  is  important  to  emphasize  that  these  gaps  are  based  on 
the  literature  we  reviewed  and  do  not  necessarily  mean 
that  the  information  is  not  known  elsewhere  within  the 
ATO.  However,  if  the  information  does  exist,  then  it  is  also 
important  to  note  that,  in  our  search  of  internal  documents 
via  the  FAA  intranet  and  personal  contacts  within  the  ATO, 
we  did  not  uncover  it.  Our  discussion  is  organized  around 
five  central  themes  derived  from  the  literature: 

1 .  Consistency  of  OE  data  collected  across  time 

2.  Consistency  of  OE  research 

3.  Evaluation  of  OE  interventions 

4.  Multiple  levels  of  OE  analyses 

5.  Multifaceted  nature  of  controller  performance 

Within  each  theme,  we  will  describe  what  we  know  and 
don’t  know  based  on  our  review  of  the  literature. 

Consistency  of  OE  Data  Collected 
Across  Time 

What  We  Know 

During  the  last  two  decades,  the  methods  used  to  conduct 
OE  investigations  have  been  relatively  consistent.  FAA 
Order  7210.56  (2002b)  describes  a  systematic  approach 
to  assessing  factors  contributing  to  an  OE.  The  major 
causal  factor  categories  in  the  OE  reporting  form  have 
also  remained  relatively  consistent.  For  example,  the  maj  or 
categories  in  the  Causal  Factor  section  of  Form  7210-3 
(data  posting,  radar  display,  aircraft  observation,  com¬ 
munications,  coordination,  and  position  relief  briefing) 
are  the  same  as  those  referenced  in  Order  721 0-47  (FAA, 
1982).  These  categories  do  not  describe  HF  issues  such  as 
controller  mental  processes  but,  instead,  describe  events 
that  occurred  at  the  same  time  as  the  OE. 

This  has  led  to  a  long  history  of  conducting  research 
and  implementing  initiatives  that  address  the  role  of 
factors  such  as  the  amount  of  traffic,  miscommunica- 
tions  (hearback/readback),  coordination,  and  position 
relief  briefings  on  OEs.  Other  variables  of  interest  from 
a  research  perspective  have  typically  not  been  collected 
or  have  only  been  collected  on  a  superficial  level:  traffic 
complexity,  situation  awareness,  and  some  temporal  and 
demographic  factors. 

Efforts  have  been  initiated  over  the  years  to  improve  the 
quality  of  some  of  the  information  obtained  during  the 
OE  investigation  process.  For  example,  instead  of  using 
a  subjective  measure  of  traffic  volume,  the  measure  is 
now  focused  on  the  number  of  aircraft  handled  by  the 


controller  at  the  time  of  the  error.  Three  questions  about 
SA  (failure  to  detect,  comprehend,  or  project  the  future 
status  of  displayed  data)  were  added  to  the  form  in  the 
late  1990s.  Other  measures,  such  as  complexity,  remain 
subjective  and  difficult  to  define  while  others  have  not 
been  addressed  because  insufficient  data  are  collected 
during  the  investigations. 

What  We  Don’t  Know 

Because  the  same  kinds  of  data  have  been  collected  over 
time,  we  tend  to  conduct  many  of  the  same  analyses, 
get  the  same  findings,  and  implement  the  same  kinds 
of  initiatives  over  and  over.  We  know  less  about  how 
other  variables  (such  as  organizational,  managerial,  and 
supervisory  factors)  that  we  have  not  historically  collected 
may  be  related  to  OEs. 

The  kind  of  information  that  has  historically  been  gathered 
during  the  OE  investigation  process  tends  to  be  descrip¬ 
tive  and  does  not  address  many  of  the  underlying  causal 
dimensions  typically  associated  with  human  error.  For 
example,  while  job  task  analyses  and  personal  reports  em¬ 
phasize  the  importance  of  memory  and  decision  making 
in  air  traffic  control,  this  factor  has  not  been  adequately 
addressed  as  part  of  the  investigation  process.  This  can 
be  attributed,  in  part,  to  the  fact  that  we  cannot  directly 
observe  how  memory  influences  the  controller’s  decisions 
and  overall  performance. 

Another  problem  with  OE  data  collection  involves 
the  way  OEs  are  identified.  While  potential  OEs  are 
systematically  identified  (and  later  confirmed)  in  en 
route  facilities  due  to  the  presence  of  the  “snitch  patch” 
in  the  en  route  HOST  software,  the  current  approach 
to  identifying  OEs  that  occur  at  TRACONs  and  tow¬ 
ers  comes  through  verbal  reports  made  by  the  involved 
controller,  a  supervisor,  fellow  controllers,  or  a  pilot.  As 
a  result  of  differences  in  how  OEs  are  identified,  we  do 
not  know  the  extent  to  which  OEs  in  TRACONs  and 
towers  are  underreported.  We  are  also  unable  to  ensure 
that  intervention  strategies  found  to  be  effective  in  one 
area  will  be  of  equal  benefit  in  others. 

Consistency  of  OE  Research 
What  We  Know 

Several  variables  have  received  repeated  attention  over 
the  last  40  years;  the  three  areas  that  have  received  most 
of  the  attention  are:  amount  of  traffic,  time  on  position, 
and  pilot/ controller  communications,  specifically  hear¬ 
back/readback  errors. 
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what  We  Don’t  Know 


Amount  of  Traffic 

The  amount  of  traffic  counted  on  a  national  basis  is  perhaps 
the  single  most  important  determinant  of  the  frequency 
of  OEs.  As  the  overall  volume  of  traffic  increases,  there 
has  historically  been  a  corresponding  increase  in  OEs. 
No  other  single  variable  has  accounted  for  more  variance 
in  OEs  than  traffic  volume,  ffowever,  the  relationship 
becomes  more  complex  when  considered  at  the  sector/ po¬ 
sition  level.  Despite  the  steady  growth  in  air  traffic  during 
thel970sandl980s,  the  percentage  of  OEs  that  occurred 
under  conditions  of  light  and  moderate  traffic  increased 
and  was  accompanied  by  a  corresponding  decline  in  the 
percentage  of  OEs  attributed  to  heavy  traffic.  During  the 
past  seven  years,  our  review  of  the  OES  database  reveals 
that  the  number  of  aircraft  handled  by  the  controller  at 
the  time  of  an  OE  has  remained  relatively  consistent,  de¬ 
spite  changes  in  the  overall  amount  of  traffic.  In  general, 
while  the  amount  of  traffic  has  increased  at  the  level  of 
the  NAS,  on  a  sector  level,  the  amount  of  traffic  associ¬ 
ated  with  OEs  has  remained  unchanged. 

Time  on  Position 

Research  has  consistently  shown  that  a  relatively  high 
percentage  of  OEs  occur  during  the  first  20  minutes  on 
position,  and  that  relationship  is  consistent  across  op¬ 
tions  and  during  much  of  the  day.  While  the  tendency 
for  OEs  to  occur  early  after  taking  over  a  position  is 
often  linked  with  position  relief  briefings,  a  summary 
of  the  NAIMS  data  from  1999  to  2003  revealed  that 
only  a  small  percentage  of  OEs  were  related  to  position 
relief  briefings. 

Communications 

Miscommunication  has  historically  been  identified  as 
a  primary  causal  factor  associated  with  OEs.  Of  all  the 
miscommunication  errors,  hearback/readback  errors 
were  most  commonly  studied.  Research  has  shown  that, 
under  normal  operations,  only  a  small  percentage  of  all 
controller  communications  involve  a  hearback/readback 
error,  and  most  of  those  errors  are  caught.  Thus,  few 
hearback/readback  errors  result  in  an  OE.  Elowever,  this 
does  not  negate  the  need  to  continue  to  conduct  research 
on  this  issue  because  it  still  is  an  important  factor  related 
to  OE  occurrence. 


Amount  of  Traffic 

We  know  little  about  the  interaction  between  the  amount 
of  traffic  and  other  factors  such  as  airspace  complexity. 
Most  of  the  research  on  airspace  complexity  has  addressed 
variables  that  predict  sector  congestion,  not  OEs.  The 
limited  amount  of  research  that  has  related  complexity 
to  OEs  has  had  inconsistent  results.  The  exception  is 
that  we  know  that  smaller  sectors  tend  to  have  higher 
error  rates.  Thus,  more  research  is  needed  that  specifi¬ 
cally  relates  airspace  complexity  to  OEs.  We  also  need 
to  develop  a  better  understanding  of  how  much  time 
controllers  spend  working  traffic  under  various  work¬ 
load/complexity  conditions. 

Time  on  Position 

Most  research  on  the  relationship  between  time  on  po¬ 
sition  and  OEs  has  focused  on  position  relief  briefings, 
even  though  they  were  infrequently  identified  as  a  causal 
factor  in  OEs.  Does  the  relatively  higher  number  of  OEs 
that  occur  soon  after  taking  over  a  position  occur  because 
most  controllers  only  work  on  a  sector/position  for  short 
periods  of  time?  Baseline  information  about  how  long 
controllers  actually  work  a  sector  or  position  is  necessary 
to  understand  this  phenomenon.  We  also  don’t  know 
how  factors  other  than  position  relief  briefings,  such  as 
readiness  to  perform,  might  contribute  to  OEs  that  occur 
soon  after  taking  over  a  position. 

Communications 

Because  most  hearback/readback  errors  are  caught, 
we  don’t  know  the  conditions  that  make  a  controller 
vulnerable  to  failing  to  catch  this  kind  of  error  during 
a  communications  exchange.  For  example,  workload  or 
traffic  complexity  might  occasionally  interfere  with  a 
controller’s  ability  to  catch  a  hearback/readback  error. 
To  identify  these  factors,  research  involving  multiple 
variables  is  required. 

Introduction  of  equipment  that  reduces  the  requirement 
to  communicate  verbally  might  reduce  the  incidence  of 
some  kinds  of  communications  errors  but  might  introduce 
other  errors  in  cognitive  processing,  especially  if  the  new 
equipment  places  a  greater  load  on  visual  processes  (e.g., 
transmission  of  text  data  link  messages  between  the  pilot 
and  controller). 
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Evaluation  of  OE  Interventions 
What  We  Know 

The  two  most  common  areas  addressed  by  national, 
regional,  and  facility  level  interventions  include  com¬ 
munications  (specifically  hearback/readback  errors)  and 
position  relief  briefings. 

What  We  Don’t  Know 

Despite  the  continued  interest  in  reducing  OEs,  we  found 
little  evidence  of  any  systematic  efforts  to  evaluate  the  ef¬ 
fectiveness  ofOE  reduction  initiatives.  While  we  received 
anecdotal  reports  about  the  success  of  certain  individual 
facility  interventions,  we  did  not  find  any  quantitative 
data  to  support  those  claims.  This  does  not  mean  that 
facility-level  data  are  unavailable;  it  merely  means  that 
these  data  were  not  included  in  any  available  OE  data 
sources  used  for  this  study.  Even  at  the  national  level 
where  a  number  of  OE  reduction  initiatives  have  been 
implemented,  we  could  not  find  any  documentation  of 
the  effectiveness  of  those  initiatives.  Without  efforts  to 
establish  a  method  to  record  and  track  the  effectiveness 
of  OE  mitigation  initiatives,  we  will  be  unable  to  learn 
from  past  successes  and  failures.  We  were  also  unable 
to  examine  strategies  that  were  claimed  to  be  effective 
at  one  facility  to  determine  if  they  could  be  effective  at 
another. 

Multiple  Levels  of  OE  Analyses 
What  We  Know 

Analysis  of  OEs  is  done  by  many  parts  of  the  organiza¬ 
tion  at  many  levels  for  many  purposes.  For  example, 
senior  management  has  often  focused  their  attention  on 
national  OE  rates  and  reduced  the  percentage  of  OEs  that 
fall  in  the  highest  severity  category.  Data  analyses  and 
initiatives  focused  at  this  level  on  information  averaged 
over  individual  controllers  and  sectors  or  positions.  On 
the  other  hand,  facility  management  generally  focused 
their  efforts  on  particular  controllers  and  sectors  that  had 
more  OEs  than  others  and  on  techniques  and  procedures 
often  thought  to  be  closely  associated  with  OEs.  Thus, 
national  initiatives  are  often  directed  towards  a  more 
generic  approach  to  reducing  OEs  while  facility  initia¬ 
tives  may  address  problems  associated  with  a  local  issue, 
such  as  airspace. 

A  number  of  tools  are  available  that  allow  us  to  conduct 
OE  analyses  to  support  questions  asked  at  different 
levels  of  the  organization.  For  example,  the  JANUS 


technique  illustrates  the  many  levels  of  analysis  that  are 
possible  if  sufficient  information  is  available  from  the 
OE  investigation.  OE  re-creation  tools  (e.g.,  SATORI 
and  RAPTOR)  provide  an  excellent  opportunity  to  gain 
additional  understanding  about  observable  conditions 
surrounding  OEs. 

What  We  Don’t  Know 

We  need  to  understand  how  various  factors  interact  to 
produce  OE  conditions  by  recognizing  that  multiple 
individual,  situational,  and  organizational  factors  both 
contribute  to  OEs  and  serve  as  a  deterrent  to  them. 
We  should  follow  recommendations  first  made  three 
decades  ago  to  expand  the  gathering  of  human  factors 
information  during  the  OE  investigation  process.  Once 
this  is  accomplished,  we  can  develop  analytic  strategies 
to  identify  how  various  factors  interact  to  contribute  to 
the  OE  process. 

To  date,  most  analyses  have  focused  on  one  or  two  vari¬ 
ables.  The  future  lies  in  efforts  to  fully  address  the  dynamic 
multi-factor  and  multi-level  conditions  associated  with 
controller  performance.  For  example,  we  know  from 
some  studies  that  supervisors  can  play  an  important  role 
in  reducing  the  likelihood  of  OEs.  Elowever,  research  has 
not  addressed  several  relevant  questions:  What  aspects  of 
their  roles  and  responsibilities  are  most  critical?  To  what 
extent  can  supervisors  affect  OEs  by  identifying  poor  per¬ 
formance,  overseeing  position  relief  briefings,  providing 
support  during  high  traffic  loads,  or  taking  other  actions? 
Until  we  gain  a  better  understanding  of  the  dynamic 
nature  of  the  OE  process  and  the  role  of  interactive 
factors,  we  are  likely  to  continue  to  experience  difficul¬ 
ties  in  developing  truly  effective  intervention  strategies. 
Finally,  we  have  to  consider  broader  organizational  issues. 
Are  initiatives  designed  to  address  conditions  within  an 
individual  facility  suitable  for  national  deployment?  We 
know  little  about  these  issues  other  than  certain  facili¬ 
ties  have  higher  error  rates  than  others.  Further  research 
is  needed  at  higher  organizational  levels  to  understand 
which  HE  causes  of  OEs  are  common  across  facilities  and 
which  are  specific  to  a  given  facility,  area,  sector,  and/or 
individual  or  team. 

The  Multifaceted  Nature  of  Controller  Performance 
What  We  Know 

Over  the  course  of  their  careers,  many  controllers  will  not 
have  an  OE,  a  small  percentage  will  have  one,  and  a  still 
smaller  percentage  will  have  multiple  OEs.  All  controllers 
are  trained  and  certified  to  control  traffic  in  a  particular 
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role  (i.e.,  position)  and  on  specific  sections  ofairspace  (e.g., 
sectors).  Certification  is  required  because  each  section  of 
airspace  is  unique  due  to  a  variety  of  factors  including 
size,  airway  configuration,  traffic  patterns,  traffic  mk,  etc. 
These  factors  interact  with  the  techniques  controllers  use 
to  control  traffic  to  create  a  unique  environment.  Despite 
the  complex  and  dynamic  nature  of  a  controller’s  tasks, 
most  investigations  have  focused  on  only  one  or  two 
factors  in  the  study  of  controller  performance. 

What  We  Don’t  Know 

We  do  not  understand  the  importance  of  the  techniques 
and  procedures  some  controllers  adopt  to  remain  relatively 
error-free.  Additional  information  is  needed  to  identify 
aspects  of  controller  performance  that  allow  some  to  re¬ 
main  “error  free”  or  relatively  error  free  throughout  much 
of  their  careers.  Are  these  skills  developed  during  training, 
or  are  some  individuals,  because  of  their  innate  abilities, 
better  controllers  than  others?  We  suggest  that  efforts 
to  identify  variables  that  affect  controller  performance 
should  be  based  on  a  cognitive  model  of  ATCS  tasks  such 
as  that  provided  in  Wickens  et  al.  (1997,  p.  93).  Using  a 
cognitive  model  to  guide  the  choice  of  variables  and  data 
analysis  should  help  us  understand  where  breakdowns 
occur  in  relation  to  certain  types  of  errors. 

We  have  not  adequately  measured  controller  performance 
during  training  and  certification.  Such  measures  would 
allow  us  to  evaluate  the  relative  importance  of  identifi¬ 
able  weaknesses  on  the  occurrence  of  future  OEs.  We 
also  have  limited  knowledge  about  the  extent  to  which 
OEs  are  influenced  by  airspace  or  traffic  complexity 
beyond  knowing  that  smaller  sectors  have  higher  error 
rates.  Because  errors  in  controller  performance  are  likely 
to  occur  a  number  of  times  and  not  lead  to  an  OE,  what 
combination  of  factors  or  conditions  prevents  an  OE 
from  occurring?  Is  it  related  to  attention,  memory,  deci¬ 
sion-making,  or  other  factors? 

RECOMMENDATIONS 

Based  on  our  review  of  the  OE  research  literature  and  the 
discussion  above,  we  recommend  the  following: 

Expand  the  OES  database  to  better  identify  the  human factors 
associated  with  OEs.  There  are  two  ways  to  improve  the 
existing  database.  The  first  concerns  the  accuracy  of  the 
information.  Continued  efforts  are  needed  to  ensure  that 
investigators’ judgments  about  subjective  categorizations, 
both  within  and  across  facilities,  are  made  consistently. 
For  example,  criteria  about  the  complexity  of  traffic  con¬ 
ditions  need  to  be  clearly  laid  out  so  that  judgments  can 


be  made  consistently.  Even  on  more  objective  measures 
(e.g.,  amount  of  traffic)  we  need  to  ensure  that  the  time 
period  during  which  traffic  is  counted  is  consistent  across 
OEs.  As  part  of  this  effort,  there  is  a  need  to  ensure  that 
individuals  responsible  for  gathering  information  dur¬ 
ing  the  OE  investigative  process  have  been  adequately 
trained.  Unless  data  are  gathered  consistently,  we  will  be 
unable  to  adequately  evaluate  differences  across  facilities. 
Another  element  about  the  accuracy  of  the  OE  database 
concerns  data  entry.  During  our  analyses  of  the  database, 
we  encountered  numerous  instances  where  data  were  in¬ 
complete  or  inaccurate.  Additional  safeguards  are  needed 
to  ensure  that  values  for  variables  in  the  database  do  not 
exceed  specified  limits.  This  is  one  way  of  improving 
data  quality. 

The  second  way  to  improve  the  database  involves  adopting 
the  repeated  recommendations  made  over  the  past  20  years 
to  include  additional  categories  that  address  HE  dimen¬ 
sions  traditionally  linked  to  controller  performance  and 
human  error.  This  information  would  more  adequately 
describe  the  individual,  situational,  and  organizational 
events  that  contributed  to  each  OE.  We  feel  that  the  cur¬ 
rent  effort  to  utilize  JANUS  to  analyze  OEs  at  selected 
facilities  is  a  step  in  the  right  direction.  By  improving 
the  investigation  process,  we  will  have  a  richer  database 
to  assess  the  human  factors  most  closely  associated  with 
OEs,  one  that  will  facilitate  our  understanding  of  potential 
linkages  between  factors  and  will  provide  greater  support 
for  efforts  to  assess  the  effectiveness  of  OE  intervention 
strategies.  Another  way  to  improve  OE  investigations 
would  be  to  adopt  the  DOT  Inspector  General’s  recom¬ 
mendation  (DOT,  2003)  to  develop  a  method  to  better 
detect  and  re-create  OEs  in  terminal  facilities. 

Obtain  baseline  information  about  air  traffic  operations  to 
better  understand  conditions  surrounding  OEs.  Baseline 
information  about  normal  AT  operations  is  needed  to 
differentiate  between  conditions  in  which  OEs  occur  and 
those  in  which  OEs  do  not  occur.  For  example,  how  does 
the  level  of  traffic  typically  associated  with  OEs  compare 
with  the  amount  of  traffic  controllers  typically  experience? 
It  would  also  be  useful  to  obtain  baseline  information 
about  the  frequency  with  which  supervisors  monitor 
controller  performance  and  the  traffic  situation. 

We  must  also  be  able  to  identify  and  track  the  introduc¬ 
tion  of  new  NAS  procedures/  technologies  and  assess  their 
short-  and  long-term  effects  on  controller  performance 
and  OE  occurrence.  Questions  have  been  raised  regard¬ 
ing  the  effects  of  URET  (Corker,  2004),  as  well  as  efforts 
to  use  color  to  differentiate  “own”  from  “other”  aircraft 
with  the  introduction  of  STARS  at  the  Philadelphia 
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TRACON.  We  also  need  to  ensure  that  the  information 
collected  during  OE  investigations  is  sufficiently  sensi¬ 
tive  to  allow  us  to  determine  how  a  new  technology  or 
procedure  affected  the  incidence  of  OEs.  For  example,  if 
shifting  entries  on  URET’s  Aircraft  List  (ACL)  distracted 
the  controller  so  that  he/she  committed  an  OE,  under 
today’s  system  the  error  might  be  linked  to  the  result  of 
the  distraction,  rather  than  its  cause.  Continued  efforts 
are  needed  to  document  these  and  other  NAS  procedural 
and  technological  changes  so  that  scientists  can  adequately 
assess  their  potential  influence  on  the  OE  process. 

Some  of  these  changes  may  dictate  different  ways  in 
which  we  should  aggregate  and  analyze  the  OE  data. 
For  example,  increases  in  direct  routing  of  aircraft  may 
have  shifted  conflict  points  and  required  controllers  to 
adjust  some  of  their  procedures  and  practices.  When  OE 
data  are  only  aggregated  by  facility,  we  may  miss  subtle 
changes  in  the  OE  process.  Greater  recognition  should 
be  given  to  the  dynamic  nature  of  air  traffic  control  and 
adjust  the  baseline  and  OE  data  analysis  processes  to 
more  adequately  address  the  new  conditions. 

Conduct  analyses  that  account  for  the  interactive  nature  of 
variables  associated  with  OEs.  Given  that  there  have  been 
few  systematic  attempts  to  understand  the  interaction  of 
the  multiple  factors  involved  in  the  OE  process,  future 
research  should  be  directed  toward  investigating  their 
interaction.  Without  this  emphasis,  we  are  not  likely 
to  benefit  from  more  sophisticated  data  collection  and 
analysis  strategies  such  as  those  suggested  by  JANUS. 

Develop  method  of trackingand  evaluating  OE  intervention 
strategies.  Few  facilities  have  tracked  the  effects  of  their 
OE  reduction  initiatives  using  measurable  outcomes. 
Furthermore,  there  are  few  reliable  indications  that 
these  initiatives  have  caused  OE  rates  to  decline.  While 
some  information  about  OE  reduction  initiatives  may  be 
available  to  the  ATO,  we  were  unable  to  document  any 
systematic  efforts  to  identify  or  assess  the  effectiveness  of 
initiatives  undertaken  to  reduce  OEs.  This  may  be  partly 
because  there  is  no  central  repository  of  information  about 
the  implementation  of  local  or  national  initiatives  or  a 
record  of  their  outcomes.  To  develop  such  an  information 
repository,  facilities  must  first  provide  a  description  of 
their  OE  reduction  initiatives  along  with  observed  results, 
and  national  offices  must  compile  information  from  the 
individual  reports  to  track  the  success  of  different  types  of 
initiatives.  Once  identified,  initiatives  that  are  successful 
at  one  facility  may  be  adapted  to  others. 

Develop  measures  of  controller  performance.  Even  though 
controllers  often  feel  that  poor  techniques  and  procedures 


on  the  part  of  their  colleagues  are  causal  factors  in  OEs, 
we  could  find  little  scientific  literature  that  adequately 
described  the  role  of  techniques  and  procedures  on  OEs. 
We  also  did  not  find  any  literature  that  described  the 
techniques  and  procedures  used  by  controllers  who  have 
remained  error-free  over  much  of  their  careers.  Efforts 
are  needed  to  identify  those  “good”  techniques  and  pro¬ 
cedures  that  allow  some  controllers  to  remain  error  free. 
By  developing  a  more  complete  understanding  of  the 
performance  of  controllers  who  do  not  commit  OEs,  we 
can  design  improved  selection  and  training  programs  to 
ensure  that  those  knowledge,  skills,  and  abilities  are  pres¬ 
ent  in  certified  controllers.  An  improved  set  of  methods 
for  periodically  assessing  operational  performance  will 
also  allow  supervisors  to  more  readily  identify  aspects  of 
controller  performance  that  need  improvement.  Some  of 
the  initiatives  have  focused  on  these  activities.  However, 
to  our  knowledge,  these  efforts  have  not  developed  stan¬ 
dardized  procedures  to  assess  controller  performance. 

Over  the  next  decade,  the  number  of  retiring  control¬ 
lers  will  lead  to  a  continuous  flow  of  new  trainees  into 
AT  facilities.  Efforts  are  needed  to  establish  a  database 
about  aspects  of  controller  training  performance  that 
can  be  linked  with  the  OE  database.  There  should  be 
sufficient  granularity  of  performance  assessments  made 
during  OJT  to  allow  scientists  to  determine  if  specific 
weaknesses  in  training  performance  are  associated  with 
subsequent  OEs.  Without  such  a  database,  there  will  be 
no  way  of  knowing  whether  training  is  effective  and  if 
adjustments  in  trainingwould  result  in  improved  control¬ 
ler  performance.  Special  attention  needs  to  be  focused  on 
the  first  fewyears  following  certification  because  evidence 
suggests  that  younger,  more  inexperienced  controllers 
have  higher  error  rates. 

Identify  new  approaches  to  reduce  high  impact  causal  factors. 
Our  review  identified  several  factors  (e.g. ,  hearback/ read- 
back,  amount  of  traffic,  time  vulnerability)  that  have 
historically  been  associated  with  OEs.  Although  previous 
error  mitigation  strategies  attempted  to  address  these 
factors,  there  is  no  evidence  that  their  influence  has  been 
significantly  reduced.  Thus,  we  continue  to  consider  them 
to  be  fruitful  areas  of  research.  However,  new  approaches 
are  needed  to  identify  strategies  that  can  reduce  the  role 
of  miscommunication,  time  vulnerability,  and  other 
factors  on  OEs.  New  investigations  should  focus  on  the 
interplay  of  several  factors  in  OE  causation. 

Given  the  current  emphasis  on  OE  severity,  we  need  to 
more  clearly  understand  the  relationship  between  OE 
severity  and  various  causal  factors.  Do  the  events  that  lead 
up  to  more  severe  OEs  differ  from  those  where  severity 
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remains  relatively  low?  If  so,  can  interventions  be  devel¬ 
oped  to  interrupt  the  chain  of  events  that  lead  up  to  the 
more  severe  OEs  and  thereby  ensure  that  either  severity 
remains  low  or  that  the  OE  likelihood  is  reduced? 

From  our  review  of  the  literature,  it  is  clear  that  if  the 
human  factors  research  community  continues  to  address 
issues  concerning  OEs  in  isolation  of  one  another  and  of 
operations,  it  is  unlikely  that  we  will  discover  more  than 
what  we  already  know.  Instead,  this  review  suggests  that  it  is 
time  for  a  more  comprehensive  and  coordinated  approach 
to  conducting  research  about  HF  causes  of  OEs.  A  national 
plan  coordinated  by  the  ATO’s  Safety  Service  Office  is 
needed  to  identify  the  respective  roles  of  members  of  the 
research  community  in  investigating  these  issues  and  work 
with  the  operational  community  to  translate  the  results 
into  a  set  of  operational  OE  reduction  initiatives. 

Examine  OEs  as  a  part  of  a  safety  culture.  It  would  be 
useful  to  consider  how  OEs  could  be  examined  from 
the  perspective  of  a  “safety  culture.”  By  this,  we  mean 
the  social  norms  and  expectations  present  in  an  air  traf¬ 
fic  facility  that  influence  the  behavior  of  operational 
personnel.  For  example,  a  perceived  punitive  approach 
(norm  or  expectation)  for  dealing  with  OEs  can  result 
in  underreporting  (behavior)  at  facilities  where  OEs  are 
not  automatically  identified.  No  data  are  available  about 
how  safety-related  norms  are  developed  in  ATC  facilities 
or  how  they  affect  behavior. 

Other  characteristics  of  a  safety  culture  include  a  dem¬ 
onstration  in  all  levels  of  the  organization  of  values  that 
support  the  goal  of  continually  identifying  and  resolving 
safety-related  problems,  proactivity  (rather  than  reactivity) 
in  searching  for  and  resolving  safety  hazards  (especially 
through  open  discussion  of  problems),  and  development 
of  metrics  and  data  collection  methods  to  assess  safety 
levels  and  identify  causal  factors  associated  with  errors. 

It  is  not  possible  to  study  a  safety  culture  unless  you  can 
measure  safety.  Currently,  the  FAA  Flight  Plan  (FAA, 
2005b)  uses  OEs  as  the  primary  measure  of  ATC  safety. 
Flowever,  OEs  may  be  inappropriate  indicators  of  safety, 
if  used  alone,  because  they  only  occur  infrequently  and  do 
not  describe  safety  conditions  in  normal  operations.  Other, 
more  appropriate  measures  might  be  better  indicators  of 
air  traffic  safety,  such  as  the  number  of  miscommunica- 
tions  between  pilots  and  controllers  or  the  number  of 
data  entry  errors  that  occur  at  a  sector  when  it  gets  busy. 
Flowever,  such  measures  are  often  difficult  to  gather  and 
may  be  influenced  by  factors  other  than  safety.  There  is 
also  the  problem  of  whether  to  measure  the  process  or 
the  outcome,  or  both. 


If  the  goal  of  a  safety  culture  is  to  collect  all  available 
information  related  to  the  safety  of  an  operation,  it  is 
necessary  to  collect  information  relevant  to  all  aspects 
of  safety,  not  just  those  related  to  when  a  safety  criterion 
is  violated.  It  is  necessary  to  be  aware  of  what  happens 
during  normal  operations  and  how  other  safety-related 
measures  change  before  and  after  a  criterion  is  violated. 
It  is  also  necessary  to  obtain  as  much  information  as  pos¬ 
sible  about  a  safety  violation,  which  requires  providing 
positive  reinforcement  to  individuals  to  encourage  them 
to  provide  information  to  help  the  organization  increase 
the  safety  of  its  operations. 

We  believe  that  the  investigation  of  OEs  from  the  per¬ 
spective  of  a  safety  culture  must  be  based  on  a  systematic 
collection  and  analysis  of  all  data  that  could  be  relevant 
to  identifying  meaningful  causal  factors.  JANUS  is  an 
example  of  a  taxonomy  that  provides  a  structure  for  col¬ 
lecting  OE  causal  factors  data  from  a  different  perspective 
than  has  been  used  before.  Instead  of  looking  at  only 
event- related  errors  such  as  those  associated  with  commu¬ 
nications,  data  posting,  or  radar  displays,  it  might  also  be 
useful  to  obtain  information  about  contextual  conditions, 
mental  processing  errors,  and  supervision  issues  present 
at  the  time  of  the  OE.  If  that  kind  of  information  were 
analyzed  in  the  context  of  normative  data  about  length 
of  shifts,  time  on  position,  communications,  amount 
and  complexity  of  traffic  typically  present  in  the  airspace, 
and  so  on,  it  might  be  possible  to  better  understand 
how  and  why  OEs  occurred.  If  information  from  an  ex¬ 
panded  OE  investigation  database  were  used  to  develop 
OE  reduction  initiatives,  then  organizational  tracking 
of  their  effectiveness  could  lead  to  an  understanding  of 
which  initiatives  would  be  useful  in  addressing  particular 
or  combinations  of  causal  factors  associated  with  OEs. 
This  approach  would  support  a  “safety  culture”  concept 
because  the  organization  could  learn  from  the  data  how 
to  continue  to  improve. 

The  research  community  can  help  the  AT  operational 
community  by  identifying  data  that  would  be  useful  to 
include  in  expanded,  safety-related  OE  databases  and  by 
conducting  analyses  using  those  data.  The  operational 
community  must  collect  the  data,  develop  OE  reduction 
initiatives,  and  track  their  effectiveness.  The  two  commu¬ 
nities  can  work  together  to  interpret  the  data  and  decide 
how  to  identify  initiatives  that  might  be  useful  to  reduce 
OEs.  We  know  that  the  amount  of  traffic  will  increase  over 
time.  If  OEs  are  not  examined  in  a  different  way,  then  OEs 
will  continue  to  increase  along  with  the  traffic.  Only  by 
working  together  can  the  two  communities  build  a  safety 
culture,  better  understand  the  causes  of  OEs,  and  work  to 
reduce  them  and  the  processes  leading  to  them. 


33 


In  summary,  this  review  points  to  the  following  recom¬ 
mendations: 

1)  Expand  the  OES  database  to  better  identify  the 
human  factors  associated  with  OEs, 

2)  Obtain  baseline  information  about  air  traffic  opera¬ 
tions  to  better  understand  conditions  surrounding 
OEs, 

3)  Conduct  analyses  that  account  for  the  interactive 
nature  of  variables  associated  with  OEs, 

4)  Develop  a  method  for  tracking  and  evaluating  OE 
intervention  strategies, 

5)  Develop  measures  of  controller  performance, 

6)  Identify  new  approaches  to  reduce  high-impact 
causal  factors,  and 

7)  Examine  OEs  as  a  part  of  a  safety  culture. 
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APPENDIX  C 


REVIEW  OF  OE  INTERVENTIONS 


The  legacy  ATS  organization  conducted  several  broad 
activities  over  a  number  of  years  that  focused  on  multiple 
areas  of  OE  causal  factors.  The  discussion  below  illus¬ 
trates  the  ongoing  effort  to  identify  strategies  designed 
to  reduce  OEs.  Note  that  attention  kept  returning  to 
particular  causal  factors  as  these  factors  were  repeatedly 
identified  in  OEs.  Although  the  formal  activities  over¬ 
lapped  in  some  ways  based  on  this  recurring  interest,  we 
treat  them  separately  in  this  document  for  the  purpose 
of  discussing  the  programs.  Following  these  descriptions, 
several  other  initiatives  are  described;  some  were  national 
initiatives,  some  were  facility-specific.  Some  are  ongoing, 
while  others  have  been  concluded.  The  initiatives  are  not 
necessarily  presented  in  chronological  order. 

OE  Working  Groups 

1986  Task  Group  Review.  In  1986  the  FAA  Administra¬ 
tor  directed  the  Federal  Air  Surgeon  to  form  a  working 
group  to  address  issues  associated  with  ATC  OEs.  The 
group,  comprised  of  representatives  from  the  Air  Traf¬ 
fic  Service,  the  Office  of  Aviation  Medicine,  the  FAA 
Technical  Center,  the  Office  of  Aviation  Safety,  and 
MITRE  focused  their  investigation  on  the  human  factors 
associated  with  OEs.  Following  their  review  of  historical 
information  and  the  existing  OE  database,  they  made 
several  recommendations.  These  included  support  for 
the  recently  implemented  Operational  Position  Standards 
program,  an  emphasis  on  the  use  of  “positive  air  traffic 
control  techniques,”  implementation  of  a  nationwide 
flow  control  program,  the  addition  of  HE  data  fields  to 
accident/incident  reports,  improved  approaches  to  using 
simulation  to  evaluate  ATCS  proficiency,  and  a  recom¬ 
mendation  to  periodically  sample  ATCSs’  operational 
proficiencies.  In  addition  to  several  other  factors  associ¬ 
ated  with  advanced  displays,  the  study  group  identified 
areas  in  which  additional  research  was  needed.  Scientists 
at  the  FAA  Technical  Center  were  given  responsibility  for 
addressing  issues  involving  memory  and  visual  scanning. 
As  a  result  of  this  effort,  three  reports  were  prepared  (Stein, 
1989a;  Stein,  1989b;  and  Stein  and  Bailey,  1994).  Stein 
(1989a)  reviewed  issues  related  to  controller  scanning. 
Stein  ( 1 9 8 9b)  and  Stein  and  Bailey  ( 1 994)  were  developed 
as  memory  guides  for  controllers. 

Operational  Errors  in  the  U.S.  Air  Traffic  System  —A  Task 
Group  Review  EAA.  (1987).  In  1987,  the  FAA  Admin¬ 
istrator  initiated  a  study  of  OEs  following  a  sudden 


increase  after  a  two-year  declining  trend.  A  task  group 
was  directed  to  study  system  deficiencies  related  to  OEs 
as  well  as  other  factors  that  impacted  safe  movement 
of  aircraft  in  the  system,  such  as  pilot-controller  com¬ 
munications,  procedures,  and  human  performance.  The 
group  was  also  tasked  to  make  recommendations  for 
corrective  measures.  The  group  consisted  of  management 
personnel  from  FAA  Headquarters,  operational  personnel 
from  ATC  towers,  TRACONs,  and  ARTCCs,  as  well  as 
representatives  from  the  U.S.  Air  Force,  Navy,  and  Army, 
NASA,  and  CAME  The  topics  covered  were:  Procedures 
and  Regulations;  Operations,  Communications,  and 
Equipment;  Human  Performance  and  Training;  and  HE 
Special  Emphasis  Items. 

Problems  identified  included:  existing  procedures  were 
not  being  followed,  ARTCC  separation  standards  were 
restrictive.  Federal  Aviation  Regulations  (FARs)  placed 
inadequate  responsibility  on  the  pilot,  lack  of  precision 
of  ATC  equipment  (i.e.,  the  J-ring),  introduction  of 
automated  aids  reduced  controllers’  memory  skills,  lack 
of  organizational  coordination  before  implementing  new 
procedures,  non-standardized  and  non-aligned  procedures 
and  QA  practices  across  facilities,  high  rates  of  OEs  for 
aircraft  in  transition  (climbing/ descending),  the  OE 
report  did  not  have  information  necessary  to  pinpoint 
causal  factors,  lack  of  opportunity  for  developmentals  to 
acquire  experience  on  the  position,  lack  of  experienced 
OJT  instructors,  ineffective  supervision,  controllers  didn’t 
recognize  when  they  begin  to  exceed  their  capabilities, 
lack  of  teamwork,  lack  of  knowledge  of  aircraft  perfor¬ 
mance  in  different  conditions,  inadequate  position  relief 
briefings,  poor  controller  memory  and  decision  making, 
failures  in  controllers’  memory  and  decision  making,  and 
failure  in  information  exchanges. 

Recommendations  were  developed  to  address  each  prob¬ 
lem  identified  under  these  topics.  These  included: 

•  increase  adherence  to  procedures  by  instilling  pride 
and  motivation, 

•  use  the  performance  management  rating  system  to 
establish  accountability  for  OEs, 

•  use  a  monetary  award  program  for  OE-free  perfor¬ 
mance, 

•  test  feasibility  of  visual  separation  for  the  ARTCC 
environment, 

•  further  study  reduced  separation  above  FL290  and 
in  oceanic  environments. 
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•  examine  diverging  course  separation  minimums, 

•  revise  FAR  requirements  to  specify  pilot’s  responsi¬ 
bilities  in  communications  with  ATC, 

•  improve  coordination  processes  between  controllers 
and  use  of  flight  strips, 

•  improve  organizational  coordination  prior  to  imple¬ 
menting  procedural  changes, 

•  improve  QA  staffing  at  facilities, 

•  improve  controller  training  and  instruction  by  better 
screening  of  instructors, 

•  use  the  FAA  Ffeadquarters’  Quality  Assurance  pro¬ 
gram  to  assess  the  Academy’s  training  program, 

•  provide  error  identification  and  tracking  capabilities 
for  terminals, 

•  increase  continuity  and  standardization  of  facility 
procedures  with  national  orders, 

•  track  new  procedures  to  identify  which  contribute 
to  OEs  versus  reducing  them, 

•  make  the  OE  report  meaningful  in  determining 
causal  factors, 

•  streamline  the  FAA  incident  reporting  system, 

•  make  proficiency  training  mandatory, 

•  improve  status  indicators  for  equipment  systems, 

•  use  simulators  for  controller  training, 

•  train  supervisors, 

•  ensure  that  any  disciplinary  actions  related  to  OEs 
are  taken  based  on  FAA  policy, 

•  establish  national  training  for  operational  supervisors, 

•  establish  an  FfF  newsletter  for  controllers, 

•  train  all  employees  in  teamwork, 

•  validate  use  of  position  relief  checklists, 

•  conduct  training  for  controllers  on  decision  making 
and  positive  control, 

•  develop  training  for  controllers  on  task  organization 
and  prioritization, 

•  develop  memory  aids  and  memory  training  for 
controllers, 

•  make  changes  to  phraseology  and  readback  proce¬ 
dures, 

•  institute  an  educational  program  for  pilots  on  various 
elements  of  pilot-controller  communications  con¬ 
tributing  to  OEs, 

•  provide  training  for  controllers  emphasizing  recogni¬ 
tion  of  reduced  alertness  and  mitigation  strategies, 

•  ensure  uniform  application  of  accepted  human 
performance  principles  in  designing  future  systems, 
and 

•  develop  a  method  to  assess  decrements  in  human 
performance  resulting  from  operational  demands. 

Although  several  themes  emerged  across  the  group’s 
recommendations  (e.g.,  improve  the  incident  reporting 
system,  and  develop  training  programs  for  both  controllers 


and  pilots  to  mitigate  certain  factors),  no  information 
is  available  about  what  actions,  if  any,  were  taken  as  a 
result  of  this  activity. 

Operational  Error  Prevention  and  Reduction  Workgroup 
(September,  1996).  A  joint  NATCA/FAA  OE  prevention 
workgroup  met  at  the  Center  for  Management  Develop¬ 
ment  (CMD)  to  review  former  OE  mitigation  strate¬ 
gies/research  and  to  recommend  changes  to  related  FAA 
Orders.  Their  recommendations  included:  Discontinue 
mandatory  decertification  of  controllers  subsequent  to 
OEs,  focus  on  the  system  rather  than  the  individual  when 
reviewing  OE  causal  factors,  create  mandatory  readback/ 
hearback  training,  engage  the  first-level  supervisor  to  a 
higher  degree  when  reviewing  OEs,  and  develop  a  new 
method  to  remove  subjectivity  from  the  OE  identifica¬ 
tion  process,  e.g.,  find  a  better  technique  than  using  a 
“rubber  ruler”  method  that  allowed  marked  variability  in 
the  assessment  of  whether  an  OE  had  occurred. 

Short-  Term  Operational  Error  Reduction  Initiative 

The  Short-Term  OE  Reduction  Initiative  focused  on 
programs  to  “improve  personnel  readiness,  awareness,  and 
communications  skills  that  should  lead  to  a  reduction  in 
operational  errors  and  deviations  caused  by  incomplete 
position  reliefbriefings,  hearback/ readbackfactors,  as  well 
as  other  misunderstandings  and  miscommunications” 
(FAA,  2002a) .  Emphasis  was  placed  on  maintaining  good 
operating  practices.  This  included  ensuring  that  facilities 
had  an  OE  prevention  plan  that  incorporated  a  “back  to 
basics”  program.  Three  focal  areas  were  included  in  this 
initiative:  communications,  position  relief,  and  distrac¬ 
tions.  No  archival  data  are  available  at  this  time  that  as¬ 
sessed  the  effects  of  these  programs  on  OE  reduction. 

Communications.  Errors  in  communications  between 
controllers  and  pilots  consistently  rank  within  the  top 
five  causal  factors  identified  in  all  operational  errors. 
This  initiative  focused  on  developing  good  communica¬ 
tion  skills  to  reduce  hearback/ readback  errors  as  well  as 
other  misunderstandings  in  communications.  Aware¬ 
ness  programs  targeting  readback/hearback  errors  were 
developed  by  headquarters  staff  as  well  as  by  regional 
and  facility  groups. 

Air  traffic  facilities  initiated  a  regular  emphasis  on  good 
communication  skills  using  random  tape  reviews  that 
highlighted  readback/hearback  errors  and  examples  of 
correct  phraseology.  During  “tape  talks,”  the  supervisor 
and/or  facility  staff  specialists  reviewed  voice  recordings 
of  a  controller  on  a  position  to  assess  communication 
performance.  Personnel  were  encouraged  to  assist  others 
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to  improve  their  communications  skills  through  positive 
coaching. 

A  thirty-seven  minute  training  video  titled  “Preventing 
Read  Back  Errors”  pKstmtA  strategies  to  reduce  the  com¬ 
plexity  of  each  communication  to  facilitate  understand¬ 
ing.  The  video  was  a  mandatory  briefing  item  shown  to 
all  controllers. 

Under  this  initiative,  the  first  National  Hearback- Read- 
back  Awareness  Month  (January,  2002)  was  established 
to  focus  on  catching  and  mitigating  this  type  of  com¬ 
munication  error.  Based  on  its  perceived  success,  this 
awareness  program  became  an  annual  event  with  January 
inaugurated  as  Hearback  -  Readback  Awareness  Month, 
which  included  an  ongoing  emphasis  on  good  commu¬ 
nications  skills,  random  tape  monitoring  to  highlight 
examples  of  correct  phraseology,  and  positive  coaching. 
Anecdotal  evidence  suggests  that  some  facilities  may 
still  occasionally  conduct  a  program  focusing  on  hear¬ 
back/ readback  awareness.  However,  because  no  tracking 
mechanism  exists,  it  is  unclear  which  facilities  are  doing 
this  or  what  the  impact  is  on  OE  reduction. 

Position  Relief.  The  position  relief  briefing  is  a  standard 
operating  procedure  designed  to  optimize  transfer  of 
position  responsibility  while  at  the  same  time  minimizing 
the  additional  workload  associated  with  the  task  of  trans¬ 
ferring  the  position.  Many  times,  the  relieving  controller 
will  accept  position  responsibility  before  being  fully  ready 
to  do  so.  In  some  cases,  the  controller  will  have  been 
uncomfortable  with  current  or  planned  actions  but  will 
still  accept  the  position  instead  of  asking  the  controller 
being  relieved  to  help  work  out  the  situation.  The  goal 
of  the  position  relief  initiative  was  to  reduce  operational 
errors  caused  by  incomplete  position  relief  briefings  by 
improving  personnel  readiness  and  awareness  in  taking 
position  responsibilities.  “Most  errors  occur  early  in  a  shift 
or  after  a  break.  Take  time  with  your  relief  briefing.  Use 
the  checklist.  Make  sure  that  the  relieving  controller  has 
data  he/she  needs  in  working  memory.  This  means  that 
they  are  actively  aware  of  it”  (Stein,  &  Bailey,  1994). 

As  part  of  their  facility  OE  reduction  plan,  all  managers 
were  required  to  emphasize  the  use  of  checklists  during 
position  relief  briefing  as  part  of  their  “back  to  basics” 
program.  They  also  had  to  validate  their  facility’s  posi¬ 
tion  relief  briefing  checklist,  ensuring  that  it  conformed 
to  requirements  of  FAA  Orders  71 10.65R  (FAA,  2006) 
and  7210.3  (FAA,  2005a),  as  well  as  provide  for  a  capa¬ 
bility  to  record  the  briefing  and  take  necessary  actions 
to  ensure  that  audio  recording  of  position  relief  briefings 
occurred.  Operations  supervisors/CICs  were  required  to 


ensure  that  position  relief  procedures  were  followed,  that 
the  checklist  was  used,  and  where  available,  ensure  that 
position  relief  briefings  were  recorded. 

All  personnel  received  mandatory  training  on  use  of  the 
position  relief  briefing  checklist  and  the  audio  recording 
requirement.  They  were  trained  and  encouraged  to  “have 
the  picture”  before  taking  over  a  position  and  to  accept 
position  responsibility  only  after  they  were  fully  aware 
of  the  traffic.  Both  the  relieving  and  relieved  controllers 
were  to  ensure  that  a  complete  transfer  of  responsibilities 
was  conducted.  The  relieved  controller  should  ensure  that 
the  relieving  controller  had  “the  picture”  before  leaving 
the  position.  In  the  past,  mandatory  overlap  periods  had 
been  established  i.e.,  5  minute  pre/post  relief  briefings. 
The  relieving  controller  and  the  controller  being  relieved 
were  to  establish  an  appropriate  overlap  period  as  part  of 
the  position  relief  briefing  that  would  ensure  complete 
transfer  of  position  responsibilities.  Some  facilities  man¬ 
dated  a  specific  amount  of  time  that  the  relieved  controller 
would  remain  on  position.  Other  facilities  adopted  this 
as  a  good  practice  and  allowed  the  overlap  period  to  vary, 
depending  upon  traffic  demands.  The  intent  of  the  latter 
practice  was  to  base  the  overlap  period  on  traffic  demand, 
reducing  the  likelihood  of  potential  distractions  associ¬ 
ated  with  multiple  personnel  unnecessarily  remaining  in 
operational  areas. 

Although  not  part  of  this  particular  initiative,  local  ad¬ 
aptations  to  the  position  relief  briefing  were  sometimes 
made  to  meet  system  needs.  For  example,  “each  URET 
CCFD  facility  was  asked  to  ensure  that  pertinent  URET 
CCFD  information  was  integrated  into  any  position 
relief  briefing  list,  whether  manual  or  electronic”  (FAA 
Order  7210.3;  FAA,  2005a).  Not  all  en  route  centers 
were  “URET  CCFD”  facilities,  so  not  all  were  required 
to  alter  their  briefing  procedures  in  this  way,  but  anecdotal 
evidence  suggests  that  some  facilities  made  other  local 
adaptations  to  their  position  relief  processes. 

Distractions.  The  goal  of  this  initiative  was  to  reduce 
control  room  distractions  so  that  controllers  could  in¬ 
crease  their  focus  on  the  operations.  For  example,  during 
periods  of  light  traffic,  controllers  can  be  distracted  from 
their  primary  duties  by  conversations  unrelated  to  the 
control  room  operations.  Under  this  initiative,  during 
light  traffic,  operational  supervisors/ CICs  were  to  reduce 
distractions  in  the  control  room  and  assure  control  room 
focus  on  operations. 

Control  Room  Management.  Effective  control  room 
management  requires  operations  supervisors  and  manag¬ 
ers  to  ensure  that  combining  and  decombining  of  sectors 
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and  positions  are  appropriate  for  current  and  forecasted 
traffic  conditions.  The  distractions  initiative  included 
an  effort  to  encourage  the  appropriate  combining/de- 
combining  of  sectors  and  positions.  Air  traffic  managers 
were  instructed  to  brief  all  operational  personnel  on 
their  expectations  that  sectors  and  positions  would  be 
combined/de-combined  based  on  current  and  forecasted 
traffic  conditions.  Operational  Supervisors  and  CICs  were 
instructed  to  actively  ensure  operational  focus  by  exercis¬ 
ing  good  judgment  and  ensuring  that  sector/  position 
openings  and  closings  were  appropriate  to  the  current 
and  forecasted  level  of  traffic. 

FAAINATCA  3-Year  Plan  for  OE  Reduction 

The  FAA/NATCA  Operational  Error  Reduction  Plan 
was  a  three-year  effort  (2002-2004)  to  prevent  and 
reduce  OEs.  It  was  collaboratively  developed  between 
the  Agency  and  NATCA  to  reverse  the  upward  trend  in 
OEs  and  focus  attention  on  those  scenarios  identified 
as  having  the  potential  for  the  greatest  risk  to  safety.  No 
archival  data  were  available  that  assessed  the  effects  of 
these  programs  on  OE  reduction. 

The  Plan  was  based  on  principles  of  continuous  com¬ 
munication  at  all  levels  about  performance  expectations 
and  feedback  about  performance  so  that  OE  reduction 
strategies  could  be  proactive  instead  of  reactive.  Initiatives 
in  the  3-year  plan  included: 

a)  Establishing  an  FAA/NATCA  coordinating  struc¬ 
ture  made  up  of  a  National  Safety  Board,  Steering 
Committee,  and  local  facility  Safety  Boards.  These 
groups  were  to  provide  periodic  communications 
to  employees  about  the  program.  For  example,  the 
Steering  Committee  planned  to  publish  an  Annual 
QA  Program  Oversight  Report.  A  review  of  the  3- 
year  plan  was  to  be  conducted  annually  to  ensure 
the  program  met  its  goals. 

b)  Cataloging  national,  regional  and  local  initiatives, 
including  those  from  areas/facilities  that  had  low 
numbers  of  OEs,  those  that  reduced  the  number 
and/or  severity  of  OEs,  and  those  that  identified/ 
corrected  problems  before  an  OE  occurred. 

c)  Empowering  local  safety  boards  to  develop  more  ex¬ 
tensive  in-depth  training  programs  using  tools  such  as 
URET,  DYSIM,  JANUS,  ETC,  and  ATTE.  Memory 
enhancement  training  and  other  programs  of  perfor¬ 
mance  enhancement  (e.g.,  the  CLT  training  prototype) 
and  self-development  were  also  encouraged. 

d)  Publishing  and  distributing  a  weekly  Quality  Assur¬ 
ance  Digest  of  OEs  to  all  facilities  as  a  mandatory  read 


and  initial  binder  item  to  heighten  OE  awareness. 

e)  Changing  the  Operational  Error  Final  Report  to 
develop  a  new  format  useful  and  practical  for  iden¬ 
tifying  the  root  cause(s)  of  OEs  (possibly  including 
JANUS,  once  development  is  completed). 

f)  Preparing  recommendations  for  a  comprehensive 
system  to  provide  a  realistic  view  of  safety  and  per¬ 
formance  of  the  NAS  as  an  alternative  to  the  current 
measures  and  separation  standards. 

g)  Establishing  a  Professional  Standards  Program  based 
on  peer  assessment  and  intervention. 

Teamwork.  This  initiative  facilitated  teamwork  by  ad¬ 
vocating  adoption  of  methods  to  clearly  communicate 
expectations,  such  as  holding  team  meetings  and  all-hands 
meetings,  having  briefings  on  current  trends  and  issues, 
and  discussing  OE  trends,  evaluations,  and  customer 
inputs. 

A  format  to  emphasize  lessons  learned  was  established.  It 
included  the  “In  The  Zone”  program,  which  distributed 
information  about  noteworthy  air  traffic  sessions,  e.g., 
excellence  in  individual/ team/ facility  performance,  and 
the  “In  The  Ozone”  program,  which  distributed  tapes 
of  operational  errors  and  near-OEs  for  the  purpose  of 
refresher  training  and  lessons  learned.  It  was  strongly 
recommended  that  training  conducted  using  the  “In 
The  Zone”  and  “In  the  Ozone”  should  involve  groups 
of  controllers  to  promote  teamwork. 

The  3-Year  Plan  also  specified  that  ATTE  training 
would  be  included  in  the  curriculum  for  new  hires.  It 
was  recognized  that  the  addition  of  new  hires  provided 
an  opportunity  to  instill  cultural  change  at  the  onset  of 
training,  to  emphasize  that  team  responsibility  is  highly 
valued.  Training  was  to  be  developed  jointly  by  both 
parties  and  instituted  at  the  Academy  and  CTI  schools. 
Some  elements  of  ATTE  were  incorporated  into  the 
Academy’s  courses  for  the  terminal  options.  Discussions 
between  the  Academy  and  NATCA  regarding  what  ele¬ 
ments  to  include  in  courses  for  the  en  route  option  have 
not  been  completed. 

This  initiative  emphasized  information  that  could  be  used  as 
refresher  training  for  all  controllers  and  targeted  to  specific 
individuals.  Use  of  both  refresher  and  skill  enhancement 
training  was  encouraged.  Techniques  included  using  pre¬ 
vious  OE  scenarios  and  re-creations  as  part  of  training, 
sharing  personal  accounts  by  controllers  as  lessons  learned, 
and  increasing  awareness  of  QA  focus  items  (both  errors 
and  good  examples)  through  facility  and  standup  briefings, 
employee  work  groups,  and  bulletin  board  information. 
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The  Year  of  Quality  Assurance.  To  facilitate  the  3-Year  Plan’s 
OE  reduction  goals,  “The  Year  of  Quality  Assurance” 
was  launched  in  2003.  Representatives  from  air  traffic 
management,  the  National  Air  Traffic  Supervisors’  Com¬ 
mittee  (SUPCOM),  and  the  National  AirTraffic  Control¬ 
lers’  Association  (NATCA)  embarked  on  the  “National 
Road  Show,”  sending  small  teams  to  air  traffic  facilities 
to  conduct  interactive  sessions  about  the  3-Year  Plan’s 
goals,  technical  training  discussions,  and  performance 
management  -  as  an  ongoing  process  of  observation  and 
feedback.  This  effort  was  designed  to  develop  a  shared 
understanding  of  the  goal  from  both  a  national  perspective 
and  incorporating  the  viewpoints  of  all  employees.  The 
traveling  team  explained  the  objectives  of  the  National 
Safety  Board,  the  Steering  Committee,  and  local  facility 
Safety  Boards.  The  team  also  illustrated  how  employees 
could  support  the  OE  reduction  initiatives. 

Identifying  Causal  Factors.  As  part  of  the  3-Year  Plan,  the 
FAA  and  NATCA  proposed  that  the  current  OE  Final 
Report  Form  was  time-intensive,  required  significant 
resources,  and  did  not  adequately  address  a  full  range  of 
causal  factors.  In  support  of  data-driven  decisions  about 
OE  intervention  strategies,  the  FAA  Air  Traffic  Office 
of  Investigations  determined  that  an  improved  method 
was  needed  to  identify  causal  factors  related  to  human 
performance.  This  effort  was  responsive  to  goals  in  the 
FAAs  1999  Strategic  Plan  and  the  1999  National  Avia¬ 
tion  Research  Plan.  The  result  was  a  technique,  JANUS, 
to  identify  OE  factors  related  to  human  performance 
along  with  contextual  and  contributing  conditions. 
JANUS  was  developed  and  tested  by  both  the  FAA  and 
EUROCONTROL  (Pounds  &  Isaac,  2001). 

The  JANUS  structure  was  described  earlier  as  part  of  the 
method  for  organizing  this  report.  However,  when  the 
technique  is  used  to  analyze  an  OE,  it  can  identify  very 
specific  factors  for  each  JANUS  category.  For  example, 
factors  of  mental  processes  (i.e.,  expectation  bias)  or  of 
teamwork  (i.e.,  the  lack  of  inter-facility  coordination) 
can  be  identified.  These  factors  can  then  be  targeted 
for  interventions  or  used  in  more  complex  analyses  to 
prioritize  factors.  The  Steering  Committee  reviewed 
the  current  OE  Form  and  JANUS  beta  test  results  to 
determine  if  a  new  format  for  the  OE  Form  could  be 
implemented  that  would  be  more  useful  and  practical 
as  an  aid  in  determining  the  root  cause(s)  of  operational 
errors.  As  part  of  this  effort,  the  JANUS  technique  was 
programmed  as  a  web-based  tool  (eJANUS)  and  began 
field-testing  at  en  route  centers. 

Based  on  the  findings  from  the  JANUS  beta  test  and 
a  field  survey  of  operational  personnel  commissioned 


by  AAT-200  (Conner  &  Corker,  2001),  several  studies 
were  initiated.  One  study,  using  data  from  Indianapolis 
ARTCC,  is  being  conducted  by  CAMI  to  examine  the 
relationship  between  both  static  and  dynamic  sector 
characteristics  and  OE  occurrence.  A  series  of  studies 
is  examining  the  role  of  Operational  Supervisors  in  OE 
prevention  to  identify  requirements,  practice,  and  “best 
practices”  so  that  strategies  can  be  developed  to  bridge 
these  if  needed. 

Traffic  Re-creations 

During  the  1990s,  the  FAA  AirTraffic  Evaluations  and 
Investigations  staff  and  Atlanta  air  route  traffic  control 
center  (ARTCC)  collaborated  with  the  CAMI  to  develop 
the  Systematic  Air  Traffic  Operations  Research  Initiative 
(SATORI;  Rodgers  &  Duke,  1993).  SATORI  graphically 
re-creates  radar  data  that  are  routinely  recorded  by  en  route 
air  traffic  control  facilities.  SATORI  enables  its  users  to 
re-create  traffic  samples  in  a  format  similar  to  what  was 
displayed  to  the  controller,  for  example,  showing  relative 
location  and  separation,  speeds,  and  headings  of  aircraft 
involved  in  an  OE.  SATORI  is  a  tool  that  can  be  used 
to  expand  proficiency  training  with  the  capability  of 
reviewing  significant  traffic  samples,  such  as  emergency 
situations,  distracting  events,  peak  periods  of  traffic  flow 
(“pushes”),  the  effects  of  weather  on  traffic  flow,  and  situ¬ 
ational  distracters,  to  promote  learning  from  them.  The 
Radar  Audio  Playback  Terminal  Operations  Recording 
(RAPTOR)  was  developed  by  AirTraffic  Services  in  the 
New  England  Region  and  is  a  similar  re-creation  tool  for 
terminal  operations.  SATORI,  RAPTOR,  and  other  re¬ 
creation  tools  are  currently  being  used  in  diverse  ways  for 
training,  e.g.,  OJT,  skill  enhancement,  refresher  training, 
and  review  of  OEs  during  investigations. 

Severity  Index 

Every  violation  of  separation  standards  provides  an 
important  opportunity  for  lessons  learned  and  system 
improvement,  although  not  all  operational  errors  share 
the  same  characteristics.  Separation  standards  and  pro¬ 
cedures  differ  depending  upon,  for  example,  the  type 
of  airspace,  weather  conditions,  type  of  aircraft,  and 
altitude.  The  U.S.  Department  ofTransportation,  Office 
of  Inspector  General  (DOT,  2000)  recommended  that 
the  FAA  Air  Traffic  Investigations  Division  tackle  the 
problem  of  modeling  and  defining  the  severity  of  OEs 
that  occurred  above  the  runway  surface  to  describe  the 
extent  to  which  applicable  separation  standards  were 
violated.  The  purpose  was  to  group  airborne  OEs  and 
thus  be  able  to  both  focus  resources  on  the  most  severe 
events  and  identify  factors  related  to  specific  categories  of 


C-5 


events.  Data  about  systemic  causes  of  OEs  could  then  be 
used  to  more  explicitly  direct  action  towards  the  preven¬ 
tion  of  future  occurrences. 

In  2001,  the  FAA  Air  Traffic  Evaluations  and  Investiga¬ 
tions  staff  developed  a  classification  system  to  distinguish 
between  airborne  OEs  based  on  characteristics  of  the 
operational  environment.  The  SI  categories  give  the  FAA 
a  new  method  to  distinguish  high  severity  OEs  from 
those  less  severe.  This  approach  allows  the  FAA  to  focus 
energy  on  OEs  having  common  characteristics.  Targeting 
efforts  towards  the  more  severe  OEs  is  a  more  effective 
use  of  available  resources. 

Training  Initiatives 

The  FAA  Air  Traffic  Evaluations  and  Investigations  Staff 
initiated  several  programs  that  focused  attention  on 
proactive  skill  building  through  performance  mainte¬ 
nance  and  enhancement  activities  rather  than  remedial 
training  programs.  It  was  anticipated  that  programs  that 
provide  regular  ongoing  support  of  skill  sets  routinely 
used  by  controllers  to  control  traffic  would  sustain  an 
already  highly  skilled  workforce  and  would  also  decrease 
OEs  when  incorporated  in  the  controller’s  day-to-day 
performance. 

Video  Briefing  Materials.  Analysis  of  OE  data  by  the  Air 
Traffic  Investigations  Division  Staff  revealed  several  com¬ 
plex  system  vulnerabilities.  Consequently,  they  produced 
a  series  of  videos  to  focus  awareness  on  recurring  factors 
found  to  occur  during  OEs.  Periodically,  videos  were 
also  produced  to  address  factors  related  to  incidents  and 
accidents.  Copies  of  these  were  sent  to  all  FAA  air  traffic 
facilities  for  use  as  briefing  materials. 

The  “Break  the  Chain”  video  illustrates  how  events,  if 
uninterrupted,  can  culminate  in  an  accident  or  incident 
and  how  attending  to  details  can  help  break  that  chain  of 
events.  The  video  “Collision  Course:  What  are  the  Odds?” 
depicts  an  actual  incident,  demonstrating  how  rare  and 
improbable  events  can  occur.  “Consequences  of  Simple 
Omissions”  is  a  compilation  of  actual  events  illustrating 
how  small  omitted  actions,  lack  of  attention,  failure  to 
follow  operational  practices,  compounded  by  poor  facility 
practices,  and  lack  of  self  discipline  or  professionalism 
resulted  in  incidents  ranging  from  operational  errors  to 
fatal  accidents.  “Preventing  Readback  Errors,  ”  describes 
strategies  to  reduce  the  complexity  of  communications 
and  chunk  information  in  meaningful  units  to  facilitate 
understanding. 


Skills  Training  —  Keeping  the  Mental  Edge.  The  National 
Air  Traffic  Professionalism  (NATPRO)  program  was 
a  new  training  approach  that  was  initially  developed 
under  the  sponsorship  of  the  FAA  Air  Traffic  Investiga¬ 
tions  Division.  The  program  is  a  personal  performance 
enhancement  program  intended  to  maintain  skilled 
performance,  rather  than  provide  remedial  training  for 
OE  reduction.  Instead  of  relying  solely  on  knowledge- 
based  training,  this  approach  integrates  the  concept  of 
“performance  coaching”  by  facility  personnel  who  are 
certified  as  coaches  (NATPRO  Cadre  Instructor,  Course 
No.  55095)  to  deliver  the  training  (NATPRO,  Course 
No.  55096)  at  their  facility.  The  program  combines 
knowledge  with  practice  so  that  controllers  can  experience 
their  own  strengths  and  limitations  and  can  gain  insight 
into  their  mental  processing  skills,  especially  those  used 
in  air  traffic  control.  These  skills  include  perception  and 
vigilance,  memory,  and  planning  and  decision-making, 
all  of  which  are  practiced  in  a  dynamic  environment.  In  a 
pilot  field  study  at  an  FAA  en  route  center,  this  program 
was  demonstrated  to  be  a  more  cost-effective,  personally 
rewarding  method  of  delivering  training  than  traditional 
proficiency  testing  or  remedial  training  programs  (Breed¬ 
love,  2004).  NATPRO  Series  1  was  deployed  at  en  route 
centers  in  2004. 

The  initial  NATPRO  program  (Series  1)  emphasized 
visual  acquisition  of  information.  The  course  content 
included  multi-tasking,  paying  attention,  scanning,  de¬ 
tecting  information,  focusing  on  relevant  information, 
being  ready  to  take  position,  working  “ahead  of  the  traf¬ 
fic,”  and  staying  “in  your  zone.”  Although  the  skills  are 
generic,  it  is  expected  that  by  improving  general  skills, 
individuals  will  demonstrate  a  corresponding  improve¬ 
ment  in  job  performance.  By  testing  themselves  against 
the  computer  and  experiencing  how  their  performance 
can  vary  in  relation  to  factors  such  as  distraction,  fatigue, 
boredom,  etc.,  participants  gain  increased  understand¬ 
ing  of  their  own  performance  and  can  develop  personal 
improvement  strategies.  It  will  eventually  be  necessary  to 
determine  empirically  whether  those  personal  improve¬ 
ment  strategies  will  lead  to  corresponding  improvements 
in  controller  performance  under  simulated  and  actual 
job  conditions. 

Initiatives  by  Air  Trafific  Managers 

In  2004,  air  traffic  managers  recommitted  themselves  to 
a  program  of  immediate  and  long-term  OE  reduction 
through  facility-level  actions  and  consistent  performance 
management  at  all  organizational  levels.  The  following 
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are  examples  of  initiatives  designed  by  air  traffic  manag¬ 
ers  and  their  staffs  to  address  facility-specific  problems. 
No  archival  data  are  available  to  identify  which  of  these 
were  executed  by  what  facilities  or  to  assess  the  effects  of 
these  programs  on  OE  reduction. 

Communications.  All  operational  personnel  were  briefed 
on  performance  management  initiatives.  Managers,  OMs, 
and  Operations  Supervisors  (OSs)  held  face-to-face  brief¬ 
ings  and  focused  on  individual  employees’  performance 
management  history,  CIC/OJTI  status,  and  disciplinary 
history.  Facility  air  traffic  management  briefed  their  facili¬ 
ties  on  expectations  about  performance  management  and 
the  link  between  FAA  performance,  facility  performance, 
and  supervisory  performance.  OMs  and  OSs  were  required 
to  provide  feedback  to  their  employees. 

Processes  were  examined  to  ensure  that  employees  received 
the  instruction  and  help  they  required  when  training 
needs  were  identified.  These  efforts  guaranteed  that  spe¬ 
cial  emphasis  items  would  be  assessed  and/or  addressed. 
Activities  would  be  communicated  and  coordinated  and 
appropriate  follow-up  actions  would  be  initiated. 

Operational  Standards.  Each  facility  was  required  to 
develop  clear  operational  standards  of  conduct  and  per¬ 
formance.  These  standards  were  tailored  to  meet  specific 
facility  needs  and  management-ensured  100%  compli¬ 
ance.  In  the  operational  areas,  for  example,  supervisors 
actively  engaged  in  directing  the  operation  (e.g.,  observing 
position  relief  briefings,  interacting  with  TMU,  staff¬ 
ing  to  traffic,  making  position  and  break  assignments), 
performance  management  (e.g.,  on-the-spot  corrections, 
teamwork),  and  managing  the  operational  environment 
(e.g.,  reducing  noise  levels,  limiting  distractions).  OMs 
and  OSs  ensured  100%  compliance  with  requirements 
for  technical  training  discussions,  operational  error  fol¬ 
low-up,  timely  response,  and  thorough  documentation  of 
all  identified  performance  deficiencies.  All  performance 
or  conduct  deficiencies  were  discussed  with  affected 
employees  and  documented  for  follow-up. 

Long-Term  Strategies.  Air  traffic  managers  of  en  route 
and  large  TRACON  facilities  used  the  existing  monthly 
telecon  to  discuss  issues  related  to  developmental  train¬ 
ing,  performance  management,  refresher  training,  per¬ 
formance  and  conduct  actions  related  to  multiple  OEs, 
and  ongoing  development  and  sharing  of  automation  to 
support  performance  management. 

Training  strategies  were  implemented  including  a  rede¬ 
sign  of  qualification  training  to  include  topics  of  traffic 
management,  operational  error  prevention,  teamwork. 


and  changes  to  the  air  traffic  environment  (e.g.,  URET). 
Refresher  training  addressed  causal  factors  trends  related 
to  operational  errors.  Developmental  trainingwas  changed 
to  include  true  simulation.  The  NATPRO  program  was 
deployed  to  all  en  route  centers. 

Teamwork.  For  a  period  of  time,  different  incentive  pro¬ 
grams  were  popular.  For  example,  “None  in  a  Million” 
HQ  awards  were  handed  out  to  facilities  having  no  OEs 
during  a  million  facility  operations.  Competitions  at 
the  facility  level  also  took  place,  with  various  incentives 
given  for  error-free  performance,  such  as  by  one  area  of 
specialization  within  a  center. 

“Best  Practices.  ”  In  2004,  Air  Traffic  developed  and  dis¬ 
tributed  a  list  of  “best  practices”  for  towers,  centers,  and 
flight  service  stations  to  emphasize  a  focus  on  “back  to 
basics”  performance.  The  list  was  distributed  as  a  “pocket 
card”  reminder.  For  example,  “best  practices”  included: 

1)  Give  and  receive  a  complete  and  thorough  position 
relief  briefing  (record  &  overlap); 

2)  Listen  to  ensure  that  readbacks  are  correct; 

3)  Ensure  that  coordination  and  communication  are 
clear,  complete,  and  correct;  and 

4)  Avoid  distractions. 

Performance  Management  Emphasis 

The  intent  of  this  initiative  was  to  make  training  proactive 
(i.e.,  administered  before  an  OE  occurs)  and  not  reac¬ 
tive  (i.e.,  administered  only  after  an  OE  has  occurred). 
All  facility  personnel  were  asked  to  plan  and  support 
performance  management  together.  To  accomplish  this, 
a  formal  process  was  instituted  for  conducting  Technical 
Training  Discussions  (TTDs). 

Technical  Training  Discussions  (TTDs)  and  an  example 
of  implementation.  A  TTD  is  a  discussion  between  the 
employee  and  his/her  first  level  supervisor  about  the 
employee’s  performance.  To  assess  a  controller’s  perfor¬ 
mance,  the  first-level  supervisor  is  required  to  conduct  a 
TTD  for  each  employee  certified  on  at  least  one  opera¬ 
tional  position.  These  discussions  are  neither  “pass/fail” 
nor  “satisfactory/  unsatisfactory;”  nor  do  they  include 
discussions  of  conduct. 

TTDs  are  not  based  on  a  single  “snapshot”  of  perfor¬ 
mance  but  instead  on  a  continuous  assessment  of  the 
employee’s  technical  performance  using  both  direct  and 
indirect  methods  and  on  the  appropriate  job  functions 
and  indicators  as  described  in  FAA  Order  3120.4.  Indirect 
methods  may  include  remote  monitoring,  tape  reviews. 
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and  SATORI  re-creations  of  traffic  samples  that  he/she 
controlled. 

In  a  TTD,  all  training  administered  during  an  OE/D  is 
reviewed  to  develop  and  direct  individualized  technical 
training  that  appropriately  addresses  identified  technical 
performance  issues.  An  employee  may  demonstrate  overall 
acceptable  technical  performance,  but  might  benefit  from 
technical  training  on  a  particular  skill  or  task.  If  no  new 
issues  are  identified,  this  too  is  discussed  and  documented. 
Reports  are  designed  to  allow  the  facility  to  identify  re¬ 
curring  and  significant  proficiency  training  needs  so  that 
effective  training  plans  can  be  developed. 

ATTD  is  not  conducted  only  following  an  OE.  They  can 
be  held  whenever  the  first-level  supervisor  identifies  an  area 
in  an  employee’s  technical  performance  that  might  benefit 
from  individualized  technical  training,  no  earlier  than  6 
months  following  the  employee’s  previously  documented 
technical  training  discussion,  no  later  than  60  days  after 
the  first-level  supervisor  assumes  responsibility  for  an 
employee  who  has  not  had  a  technical  training  discussion 
documented  during  the  previous  6  months. 

Houston  Center’s  Quality  Evaluation  Program  (QEP). 
In  October  2003,  Elouston  Center  developed  and 
implemented  a  Quality  Evaluation  Program  (QEP)  to 
improve  the  assessment  of  controller  performance  us¬ 
ing  Technical  Training  Assessments  (TTAs)  and  TTDs. 
The  QEP  provides  a  number  of  different  ways  to  look 
at  performance.  As  a  result  of  the  program,  most  areas 
of  specialization  showed  a  significant  improvement  in 
performance  proficiency.  Facility  QA  staff  expected  that, 
since  performance  is  directly  tied  to  operational  errors 
and  deviations  (OE/ODs),  improvement  in  individual 
and  team  performance  would  help  reduce  the  number 
of  OE/ODs. 

The  cornerstone  of  this  program  was  the  establishment 
of  two  Quality  Evaluation  Supervisor  (QES)  positions  in 
the  facility.  These  supervisors  are  assigned  to  the  Quality 
Assurance  Office  and  conducted  QEP  assessments  of 
controller  performance  independent  of  technical  assess¬ 
ments  conducted  by  the  individual  employee’s  Operations 
Supervisors.  The  QES  adhere  to  a  strict  standard  of  evalu¬ 
ation  for  each  assessment.  Items  assessed  include  altitude 
assignment  and  confirmation,  monitoring  of  readbacks, 
traffic  advisories,  route  clearances,  speed  control,  vector¬ 
ing,  beacon  assignments,  and  point  outs.  An  evaluation 
of  the  process  showed  that  the  QES  assessments  were 
accurate,  consistent,  and  objective. 


All  deficiencies  are  addressed  as  quickly  as  possible  by 
the  first-level  supervisor.  In  the  event  that  a  supervisor 
was  unable  to  address  a  particular  assessment  within  1 5 
days,  notification  was  given  to  the  Quality  Assurance 
Office  so  that  tapes  and/or  SATORI  re-creations  could 
be  made  and  retained  for  review. 

An  improved  system  for  tracking  technical  performance 
trends  was  also  developed.  The  QESs  maintained  a  master 
schedule  of  TTAs  and  TTDs  and  an  electronic  folder  for 
each  employee.  The  master  schedule  ensured  that  each 
employee  received  aTTA  every  other  month  and  a  TTD 
every  six  months.  Any  necessary  follow-up  assessments 
were  accomplished  during  the  intervening  months.  In 
an  “After  Care”  program,  the  QESs  performed  random 
TTAs  for  employees  involved  in  recent  operational  er¬ 
rors/  operational  deviations/ incidents. 

Formal  organizational  recognition  of  good  performance 
was  also  part  of  the  program.  If  an  employee  performed 
without  a  deficiency  during  aTTA,  formal  recognition  was 
awarded.  The  recognition  program  included  certificates, 
monthly  posters,  and  inclusion  on  a  Regional  Elonor 
Roll.  “Fantastic  Feats”  was  the  first  program  employed 
for  this  purpose;  however,  to  maintain  interest  and  spark 
involvement,  the  program  changed  periodically.  The 
“Chain  of  Excellence”  Program  recognized  all  deficient 
free  performance  as  a  “Link  in  the  Chain.” 

The  QES  supervisors  and  the  Quality  Assurance  (QA) 
Manager  metwith  the  OMs  on  a  regularbasis  to  discuss  all 
aspects  of  the  program  and  make  adjustments  as  needed. 
The  QES  assessments  were  tracked  so  that  results  could 
be  examined  at  the  level  of  the  individual,  specialty,  and 
facility.  All  performance  deficiency  trends  were  incorpo¬ 
rated  into  current  and  future  training  programs. 

In  addition  to  the  finding  that  most  specialties  showed  a 
significant  improvement  in  controller  performance,  TTA 
data  indicated  that  technical  performance  deficiencies 
decreased  an  average  of  16.6%  in  the  facility  and  that 
deficiency-free  performance  increased  by  an  average  of 
50.8%  over  the  same  period. 

These  data  can  also  be  used  to  produce  various  types 
of  facility  information.  For  example,  although  the  ma¬ 
jority  of  specialties  showed  a  significant  improvement 
in  performance  during  the  time  period  reported,  one 
specialty  showed  an  increase  in  deficiencies.  However, 
this  was  attributed  to  an  increase  in  the  number  of  items 
being  assessed  and  the  presence  of  heavy  weather  for  the 
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maj  ority  of  one  assessment  period.  The  overall  number  of 
deficiency-free  performances  increased  for  the  same  time 
period.  Also,  individual  performance  found  to  be  deficient 
during  other  assessment  periods  showed  improvement 
in  later  assessments.  In  sum,  the  improved  system  for 
tracking  technical  performance  trends  permitted  per¬ 
formance  deficiencies  to  be  addressed  through  quality 
training  in  a  timely  manner  both  for  the  individual  and 
for  the  facility. 

Kansas  City  Center’s  Performance  Management  Program.  In 
late  2002,  Kansas  City  ARTCC  initiated  a  Performance 
Management  Program  (PMP).  The  PMP  program  was 
based  on  the  belief  that  the  best  OE  initiative  is  directed 
at  the  level  of  the  individual  rather  than  at  groups.  The 
program  elements  included  a  Quality  Assurance  Review 
(QAR),  a  direct  observation  module,  and  a  TTD.  The 
QAR  is  a  process  conducted  for  other  types  of  system 
deficiencies  unrelated  to  OEs.  One  element  of  a  QAR 
is  a  review  of  the  employee’s  performance,  along  with 
other  factors,  such  as  procedures  or  equipment,  which 
may  have  contributed  to  a  deficiency  in  the  control  of 
traffic.  QAR  activities  may  range  from  a  discussion  of 
the  situation  with  the  involved  controllers  to  reviewing 
recorded  radar  and  voice  communications  data. 

All  forms  used  for  these  processes  are  automated  and 
linked  to  a  reporting  feature.  For  example,  the  QAR 
module  and  the  direct  observation  module  automatically 
feed  the  TTD  program.  This  process  resulted  in  very  few 
TTDs  being  conducted  without  including  individual, 
specific  performance  information  in  the  review.  This 
information  is  powerful  when  reviewed  by  the  supervisor 
with  the  controller.  For  example,  the  controller  and  the 
supervisor  can  review  a  sample  of  voice  tapes  representing 
a  controller’s  communications  on  position  over  a  period 
of  several  weeks.  They  provide  a  means  for  the  controller 
to  review  normal  phraseology  habits.  It  is  an  opportunity 
for  the  controller  to  determine  whether  good  or  poor 
habits  have  been  developed.  This  assists  the  controller 
to  enhance  his/her  communications  skills. 

After  initiating  the  program  in  April  of 2003,  the  center’s 
OE/ODs  declined  for  the  next  three  years,  resulting 
in  a  2002  “commendable”  rating  from  the  national 
evaluation  team.  Table  1  (Truelove,  Elatem,  &VanDyne, 
2005)  shows  the  number  of  OEs  by  year  of  program 
implementation. 

Computer-Based  Instruction  in  Human  Factors.  Respond¬ 
ing  to  the  push  to  reduce  OEs,  personnel  in  the  Great 
Lakes  Region’s  Air  Traffic  QA  office  distributed  a  three- 


Table  1 .  ZKC  OEs  by  year  of  PMP  implementation. 


Program  year 

Date 

#OEs 

Prior  year 

1/1/01  -  12/31/01 

29 

First  year 

1/1/02-12/31/02 

27 

Second  year 

1/1/03-12/31/03 

19 

Third  year 

1/1/04-12/31/04 

20 

Fourth  year 

1/1/05-7/31/05 

10 

module  program  on  human  factors.  FAA  headquarters 
staff  provided  the  program  to  interested  facilities  na¬ 
tionwide.  The  material  covered  topics  such  as  attitude, 
cooperation,  communication,  organization,  efficiency, 
perception,  memory,  and  attention.  Most  of  the  material 
appeared  to  be  based  on  generally  accepted  psychological 
and  physiological  theories. 

An  individual  knowledgeable  in  both  human  factors  and 
ATC  reviewed  the  material  and  expressed  the  opinion  that 
controllers  would  find  the  lessons  entertaining  but  that 
the  material  would  have  limited  training  value.  Without 
supporting  data  we  don’t  know  whether  the  materials  can 
be  applied  directly  to  controller  performance.  Take,  for 
example,  statements  such  as  “controllers  who  recognize 
and  maximize  situational  awareness  skills  automatically 
distance  themselves  from  high-risk  situations  and  errors.” 
While  this  statement  might  be  true,  it  is  not  clear  whether 
this  observation  about  automatic  behaviors  was  based 
on  any  data  and,  if  so,  whether  they  were  collected  from 
operational  controllers.  Similarly,  statements  suggesting 
that  controllers  select,  process  and  remember  only  those 
perceptions  on  which  attention  has  been  focused  (i.e.,  in 
focal  attention)  ignores  what  is  known  about  how  people 
also  process  information  that  is  not  the  focus  of  atten¬ 
tion  or  information  of  which  they  are  not  fully  aware. 
This  emphasis  on  focal  attention  might  lead  controllers 
to  discount  other  processes  that  can  also  lead  to  human 
error.  Suggestions  that  controllers  have  a  physiological 
structure  that  allows  them  to  know  the  difference  between 
important  and  unimportant  information  seem  to  be  ir¬ 
relevant  or  misleading.  Examples  given  of  important  and 
unimportant  information  are  of  a  fire  versus  a  lawn  mower. 
The  authors  suggest  that  somehow  the  physiological  brain 
structure  knows  the  difference  between  what  is  important 
and  what  is  trivial  information.  Such  broad  statements 
risk  giving  controllers  unwarranted  confidence  to  rely  on 
their  physiological  structure  as  a  form  of  performance 
“backup”  system.  Finally,  the  statement  that  the  complex 
tasks  of  air  traffic  control  require  faster  processing  speed 
to  complete  operations  should  also  include  a  discussion 
of  experience,  skill,  etc.  and  their  effect  on  performance. 
Thus,  there  are  a  number  of  concerns  associated  with  the 
development  and  use  of  this  training  module. 
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Sheridan’s  Analysis  of  OE  Reduction  Efforts.  Sheridan, 
(1997)  analyzed  the  effectiveness  of  OE  reduction  efforts 
conducted  in  the  Great  Lakes  Region  during  fiscal  year 
1996  (FY96).  At  the  beginning  of  FY96,  several  initia¬ 
tives  were  initiated  by  regional  headquarters’  managers  in 
an  attempt  to  reduce  OEs.  These  included  assigning  an 
OE  quota  to  field  managers,  not  to  be  exceeded  within 
each  air  traffic  manager’s  area  of  responsibility.  Air  traffic 
managers  were  held  accountable  through  negative  per¬ 
formance  evaluation  ratings.  For  his  analysis,  Sheridan 
used  data  from  both  OEs  and  operational  deviations 
(which  accounted  for  approximately  ten  percent  of  the 
total  error  rate).  Results  showed  that  intervention  efforts 
had  a  mixed  effect  on  OE  rates. 

•  The  en  route  OE  rate  exhibited  a  modest  correla¬ 
tion  with  traffic  volume  (r  =  .497).  Traffic  explained 
only  approximately  one-quarter  of  the  variance  in 
the  OE  rate. 

•  The  terminal  OE  rate  also  exhibited  a  modest 
correlation  with  traffic  volume  (r  =  .460).  Traffic 
explained  roughly  21%  of  the  explained  variance 
in  the  OE  rate. 

Error  rates  rose  marginally  in  the  en  route  environment 
while  a  modest  improvement  was  realized  in  the  terminal 
environment.  Data  revealed  that  en  route  OE  rates  rose 
from  8.50  per  month  to  9.83  per  month.  By  contrast, 
tower  OEs  on  average  fell  from  5.78  per  month  to  5.0 
per  month.  One  possibility  for  the  differences  lies  in  the 
changes  in  traffic  volume  experienced  during  the  different 
reporting  periods.  However,  analyses  revealed  that  traffic 
volume  explained  a  modest  percentage  of  the  explained 
variance  in  OE  rates. 

Runway  Safety  Initiatives 

In  October  2002,  after  an  accident  involving  arriving  and 
departing  aircraft  at  a  municipal  airport,  the  FAA  adopted 
NTSB  recommendations  (FAA,  2002b)  and  required  air 
traffic  control  tower  facility  managers  to  include  local 
procedures  in  their  facility  directives  to  assist  the  ground 
and  local  controllers  in  maintaining  awareness  of  aircraft 
positions  on  the  airport.  Procedures  were  adjusted  to  ad¬ 
dress  the  needs  of  a  specific  facility  or  airport. 

The  Runway  Safety  Office  identified  several  initiatives 
for  controllers  covering  topics  related  to  communications 
and  teamwork.  CBI  Modules  for  ATC  were  developed 
by  the  Runway  Safety  Office  on  topics  of  awareness  of 
conditions,  procedures,  rules,  and  tools.  The  Memory 


Enhancement  Project  convened  a  workgroup  to  review 
availability  of  off-the-shelf  programs  and  to  identify 
memory  aids  currently  used  at  facilities.  Originally,  the 
results  were  intended  to  be  distributed  to  other  facilities 
for  their  consideration  as  potentially  useful  tools.  However, 
distribution  was  not  completed. 

In  2001,  a  booklet  entitled  “Runway  Safety:  It’s  Every¬ 
body’s  Business”  was  distributed  (Cardosi,  2001).  Half 
of  the  material  was  directed  at  controllers  to  describe 
what  they  could  do  to  improve  surface  safety.  The  other 
half  of  the  material  was  directed  at  pilots  and  their  role 
in  surface  safety.  Topics  included  causes  of  OEs,  memory 
strategies,  how  to  improve  pilot-controller  communica¬ 
tions,  the  importance  of  teamwork,  and  strategies  to 
mitigate  fatigue.  Additional  recommendations  include 
the  following.  Only  recommendations  with  an  ATC 
focus  are  listed  here. 

Controller  Communications.  Three  recommendations 
were  made. 

•  Minimize  controller-pilot  voice  communications, 
e.g.,  eliminate  unnecessary  elements  of  controller 
transmissions  and  explore  expanded  use  of  standard 
(classified)  taxi  routes. 

•  Develop  operational  protocols  for  ATC  communica¬ 
tions  in  airport  operations,  e.g.,  a  requirement  for 
pilots  or  ground  personnel  to  use  standard  phraseol¬ 
ogy  when  communicating  with  ATC. 

•  Implement  improved  communication  technology  to 
reduce  blocked  or  partially  blocked  transmissions. 

AT  Procedures.  One  recommendation  was  made. 

•  Improve  procedures  for  taxiing  aircraft  into  position, 
holding  on  the  runway,  and  crossing  intersecting 
runways. 

Controller  Training.  Three  recommendations  were 
made. 

•  Promote  training  in  communications  and  coordina¬ 
tion,  such  as  cross-operational  training  for  pilots  and 
controllers  on  the  others’  duties  and  concerns. 

•  Provide  controllers  with  refresher  training  (e.g., 
phraseology,  memory  enhancement,  scanning)  for 
surface  operations. 

•  Implement  teamwork  training  to  improve  com¬ 
munications,  cooperation,  and  operations  planning 
between  AT  and  airport  users. 
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Regional  Runway  Safety  Initiatives.  Several  regional  air 
traffic  and  runway  safety  offices  developed  additional 
materials  and  programs  to  reduce  OEs,  including  a  num¬ 
ber  of  initiatives  implemented  by  the  Regional  Runway 
Safety  Program  Managers,  such  as  pilot  briefings,  Run¬ 
way  Safety  Action  Team  (RSAT)  meetings  with  airport 
operators,  tenants,  and  tower  personnel,  and  initiatives 
with  AO  PA,  such  as  issuance  of  publications  on  runway 
safety,  videos,  etc.  All  of  these  efforts  were  geared  to 
minimize  the  possibility  that  pilot  actions  would  result 
in  incidents,  including  OEs. 

A  real-time,  human-in-the-loop  simulation  study  (called 
ACTIVE- 1)  was  conducted  at  Anchorage  Tower  by  the 
FAA  Alaskan  Region  Runway  Safety  Office  to  determine 


whether  a  high  fidelity  tower  simulator  could  be  useful  to 
enhance  controller  training  (Racine  &  Sierra,  2004).  In 
particular,  the  study  investigated  whether  a  high  fidelity 
tower  simulator  would  improve  controller  recognition  and 
management  of  common  factors  leading  to  runway  incur¬ 
sions  (RI),  and  improve  their  skill  in  taking  appropriate 
actions  to  mitigate  unanticipated  events.  The  study  used  a 
series  ofoperationally  challenging  scenarios  and  a  state-of- 
the-art  tower  simulator  to  determine  if  fully  qualified  and 
current  tower  controllers  would  benefit  from  simulator 
practice.  Performance  improvement  from  pretest  to  post¬ 
test  was  based  on  measures  of  efficiency  and  safety.  Results 
showed  significant  improvements  in  detection  of  specific 
scripted  errors  as  well  as  general  trends  of  improvement 
in  other  system  performance  measures. 
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